Well.. since agreeing to subjugation costs such a large chunk of Honor (often taking them below 25 Honor), then that's one reason they won't agree.
In the three separate cases I've experienced lately, their armies were completely eliminated and all their provinces were under siege, them being down to three or less still owned (& sieged). So their Honor situation had to be the case, as they would happyily agree to admit defeat and send a hostage, but refused to swear fealty. As you can see in my screenshot above, agreeing to subjugation would've likely taken Otomo just barely below 25 Honor and thus he says "no".
I will try to find a similar case in which the defeated and besieged enemy, with only a handful of provinces left, has enough honor to keep him above 25 when he agrees and see if that works.