Don't do that crazy mistake again! If you really want to assualt a level 4+ fort, make sure to breach its walls first. Then 100k-500k infantry might be enough to take it down (depending upon the fort level).
General tips when sieging effectively is to always block ports of coastal provinces when sieges, use high number of cannons in the stack,25k cannons for 5 level forts for example. Try to reroll generals when you have high AT to get some good siege generals. I once had 4 siege general in early game as Granada & believe me he took over enemy forts in less than 60 days most of the time ! Not only stacking high number of cannons/ generals make it more possible for siege to progress but they also make siege start at a high % siege.