It is not a supported feature so there's no in-game way to do it. You need to edit the settings.txt file and change fullScreen=yes to fullScreen=no.How do I run EU3 in windowed mode? Would be handy to check manual/wiki/guide on things I don't know while in-game.
Nothing happens with Alt-Enter and I couldn't find it in game options either.