So the patch DID something, and while I still had some lag, it was much better.... for some time. I played more, and the lag came back (in the same playthrough than before).
Something I notice, clicking on a fleet, even a single ship fleet will make my FPS plummet in late game. Really it's atrocious. Even on the star map. On the other hand, watching the same fleet from up close is much less a resources drain.
Also, and I think it might be important to note, clicking on a sector controlled ship will not add lag, it's only for non sector controlled ship (ours, kinda).
I'll add my save, it's late game, technically I already won, but I wanted a late game crisis

Been lagging for some time now.