Yeah that's what I've been trying to do. Problem is, a lot of those factories seem to be pretty unprofitable during peacetime, meaning I have to subsidize them if possible, or let them close and re-open them during war time, which is a little pricey.
In my latest games, I have started to supply my army at least 65% ( 100% if I can afford it ), so that I have at least some artillery factories left when war starts. Since you always buy first from your own factories, your own demand keeps those factories alive. Also if you can sphere countries that have decent armies, that helps too, because they will buy from your domestic market first. When war starts, your factories can hire new people and increase production if possible. If you set your own peace time supply very low, your own military factories struggle really badly in the world market , because Great Powers usually have enough military factories to supply their own, and their sphere's armies, at that normal 30% peace time level, so there's really low demand.