Adventurers strike me as a particularly gamey, unrealistic and unrewarding part of the game as is.
Since you know when the bastard's coming, have your whole army (or at least enough to beat him) raised and at full morale by that deadline.
IMO adventurers should be more politically active, not just some random dudes(though they have their place as well). For example, when you take away somebody's lands, they could make marriage alliance or some sort of pact with another realm and press their claim with the backing of, let's say France or various nobles from France. And maybe even your own vassals could side with the adventurer, if they dislike you enough. This way the threat they pose would be more varied and interesting.
Also they shouldn't be automatically imprisoned. I'd much prefer if some of them escaped and started planning another attack and passed on their claims, so you would get a line of pretenders attacking you every generation or so. Like the Blackfyres, to take a fantasy example.