In my Nantes game, I just took all of Brittany at the beginning. With your event troops (even the diminished troops from the last patch) you should be able to easily beat Brittany. Just wait out the holdings (don't waste your troops on a direct assault). After I took the first county, I then kept breaking truces to take all the rest.
As for why you couldn't declare an invasion:
My best guess is the civil war made Brittany an invalid candidate for invasion. IIRC, an invasion needs to target someone with between 10-40 holdings in a kingdom. If there is a civil war, this could have reduced the target to less than 10 holdings.
I don't quite know for sure whether the holdings requirements is checked only once when you prepare the invasion or if it also checks when you are ready to declare the war.
In one of my games, one of my vassals started to prepare an invasion against Frisia (I had already taken most of it). Frisia was down to the county of Holland and the duchy of Gelre (3 counties). I Holy Warred Frisia to take Gelre, bringing it down to 1 county in Frisia (and 1 in Germany and 1 in Lotharingia). The vassal never ended up completing his invasion, presumably because Frisia now only had 1 county (4 holdings) in Frisia.