I agree, but things like castle towns should not be buidable but rather automatically upgrade as the prosperity of the county increases with the current system it is almost like you are buying population which is weird.
I disagree with this part, only because the AI is not set up to handle it. Additionally, insofar as ducats represent labor as a resource, the Lord would have to allocate that to infrastructure development, to create the physical expansion necessary.
Events for prosperous counties (especially AI ones) that raise the Castle Town level? Sign me up.
Totally removing the ability to directly invest in attracting a larger peasantry? No thanks. Better to fix this, than to scrap it.