EU4 - Development Diary - 9th of April 2019

  • Crusader Kings III Available Now!

    The realm rejoices as Paradox Interactive announces the launch of Crusader Kings III, the latest entry in the publisher’s grand strategy role-playing game franchise. Advisors may now jockey for positions of influence and adversaries should save their schemes for another day, because on this day Crusader Kings III can be purchased on Steam, the Paradox Store, and other major online retailers.


    Real Strategy Requires Cunning
Showing developer posts only. Show all posts in this thread.

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
Bonjour everyone! It is I, Le French Paradox and I will be your host for this Dev Diary.

Today we will talk about Tech Debt. As @DDRJake has been saying, tech debt has been an important focus for the EU4 team since the release of 1.28.3, the latest support patch we did for the Spain update. While our designers are thinking very hard about the yet-to-be-announced big expansion that is to come, we programmers have been busy tidying up the EU4 codebase.

tidying up.jpg

Microsoft should team up with Netflix, I would watch the hell out of this show

Jokes aside, we keep talking about Tech Debt but realize it might be a bit confusing, especially for our non-programming crowd.

While EU4 was announced at GamesCon 2012, the actual development started a year before, which makes the code 8 years old. And of course, the team at the time didn't reinvent everything, bits were pulled from EU3, some of which are still in use today. I'll let you speculate which is which in the comments.

As I'm sure you all know, technology is a rapidly evolving thing, and software engineering is no exception. In 10 years the development practices have evolved, both because the industry keeps learning and inventing new ways to solve problems, and because the ecosystem (the hardware and software that runs our games) also changes.

10 years ago, SSDs were not that widespread, 1080p was pretty new, DirectX 10 was all the rage in previews and most people were so hostile to Windows Vista that they kept running XP in 32 bits.
That is not to say that EU4 was using only stuff from the late 2000s or early 2010s until now. Over time our engineers have done their best to keep the game up to date, but some stuff inevitably slips through the cracks. It's the accumulation of all this aging stuff that we call tech debt(*).

Here's a few examples of those things that we have tackled so far.

64 bits
As the BattlePope already explained in a past DD, 32 bits was a reasonable choice when EU4 development started, but today 64 bits in the norm.
With that done EU4 will keep running with the next update of MacOS (which removes 32 bit compatibility altogether) and also be able to use more RAM. While there are no immediate plans to increase the memory usage of the game significantly, mods that add a lot of provinces and tags should be able to break the previous ceiling.

Rendering
It is no secret that our games do not use cutting edge graphics. True to its board game heritage, Europa Universalis is mostly about showing a world map with some cool dudes (and elephants) fighting on top of it. Yet if you take a look at the Imperator preview, you will see that our graphics experts have learnt a trick or two since EU4's release.

2xnk41.jpg


While Boromir is (was?) right, we still managed to replicate some of the improvements they have made. The main one is that the colorizing of the provinces on the map is now done mostly on the GPU, while in the past it used to consume precious CPU cycles to display whatever horrible experiment in bordergore your game was about.

capture.PNG

Let's play 7 differences, can you spot what changed from the current release in this beta screenshot?

Crash reporter
Our game never crashes, it just tactically exits. Yet when it happens, we want to know all about it. It helps our QA to reproduce the issue and our programmers to isolate the piece of code responsible and fix it.
Since EU4's release our engine team has made a new and improved crash reporter that brings us more data and help find problems faster. First of all, it now works on Linux and MacOS, meaning we will be able to investigate issues on those platforms much easier. Secondly, it allows us to add some metadata to the dump, such as the current year, the list of enabled mods or how many 6/6/6 heirs died to a hunting accident, helping us understand what triggered the issue.

Startup time
Improving the loading time of the game is something we always want to do. While we can't really apply the best fix to the issue (discontinue Windows support, the game just starts so much faster on Linux and MacOS, trust me), we managed to find some things we could do.
The most notable one was the upgrade of PhysFS, a 3rd party software that a lot of videogames use to load all resources (files on disk, mods, DLCs...). While still far from Unix performance, it should shave off a couple seconds from the startup time on Windows.

capture.PNG

Experienced Byzantium players recommend playing on Linux or Mac due to the very fast restart time

General performance
The speed of the game is, of course, always a concern. Every night, we run a couple games on benchmark machines and always make sure that the averages are below a certain value (80ms per in-game day is our current high threshold).
While you may think that performance would improve over time (due to new hardware for example), in practice the average usually go up during the development phase of a new expansion, and is then worked upon to bring it back to acceptable values.
Contrary to popular thought, adding new features in the game is not the only (or even main) source of performance regression. In fact, one big factor is simply the addition of tags and provinces.
Consider this: for every two countries in the game, a bunch of stuff needs to be computed each in-game day (relations, AI attitude...). For every two provinces in the game, the game has to know the way from A to B (what we call pathfinding). This is known as a quadratic problem. For those who don't favor math, it means that for X provinces, the problem complexity is X squared. Doubling the number of provinces does not double the number of computations needed, it quadruples it. Over the years, part of our job has been to allow EU4 to grow from around 2000 provinces to more than 4000 provinces.
Keep in mind though that in software engineering an optimization that makes something 0.1% faster is considered pretty good, 1% excellent and 10% probably means you introduced a serious bug (or fixed a very serious one). How do we manage to improve anything with those numbers? Simple: all those add up in the end.

That's all for today! Should you like to know more meaty details of the tech stuff, you can check-out my blog where I get into more practical details about development practices.

Next week we should return to a more classic Dev Diary by our designers, unless you can help me convince Jake to revamp the wine trade good to include grape kinds and soils and finally settle which of the Burgundian or Bordelaise tradition of winemaking is the best.

(*) It's a bit more complicated than that but should be enough for the sake of this diary
 
Last edited:

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
Will the new update add multi-thread support for eu4?
There is already.The performance is due to other factors.
Multithreading is indeed already baked in.
The thing is, not everything can be multithreaded.
For example a lot of computations in the daily update are what we call "order dependent": the final result depends on the order of execution.
Running that in parallel (through multithreading) will make the result non-deterministic, meaning two computers running the same update will get different values. In multiplayer, this would out-of-sync immediately.
 

Duplo

Rookie GameDev
Jan 26, 2018
114
17
Wait? So I've always been embarrassed for using my Mac to game with, but it's actually better??

Not better, in-game performances are usually pretty inferior on Mac, sadly. But startup time... god... I can launch and close the game three times on my Macbook before a gaming PC will be able to enter the main menu. So the strategy is: use Mac or Linux to kickstart your Byzantium and then once you consolidated your starting point move on Windows ;)
 

Duplo

Rookie GameDev
Jan 26, 2018
114
17
And the Linux performance are good compared to Windows or this is less good than Windows?
Thanks for any reply about this.

As far as I know performances on Linux are on par with Mac ones.

Do you know why it is that when mods are loaded the first time it seems to take forever but it the same mod list is loaded a second time it runs much faster? Or is that just what it feels like to me?

There's a lot of caching work when you load a new mod (or when you load the game for the first time after an update. So yes, it's much faster after the first time, you are right ;)
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
Whenever I go to load into a game and then have to exit to main menu the game completely restarts - as in it exits to desktop and then reloads.
Will you guys fix the "Return to Menu" problem? It's quite annoying that we have to restart the whole game just to get back to the menu. ((No offense, just asking
@MatRopert will next patch fix thing with "press back from nation select screen" and game restarts and loads anew? It is infuriating.
The "return to menu" is a legacy from EU3 in which going back to the menu after you start playing will result in a different checksum.
In EU4 it was quickly addressed by the "exit and restart to menu".
We have been wanting to fix it entirely for quite some time now but I can't make any promises as the cost is expected to be high and we have to prioritize it along many other things we could also look at (and, of course, the gameplay content of the next big patch).
Then again we also thought that we would never manage to make hotjoin work and in the end we did, so who knows?
 
Last edited:

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
Hmm... Did the hatching on different provinces line up before? (like HRE provinces not under HRE control). Apart from that I'm not sure I can spot any difference.
You're getting warm but not exactly there yet.
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
Why does the game have to compute pathfinding between every pair of provinces every day?

With the exception of things like canals, aren't the provinces and distances fixed and unchanging throughout the game?
We don't recompute everything every day, but we can't precompute either and need to recheck frequently because the path from A to B depends on military access, forts and other things that change from day to day and country to country.
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
I assume that much of this has already been optimized, so I'm probably not saying anything new. But wouldn't it make sense to mark out the days where there is an actual status change in military access, forts, or whatever, and only compute pathfinding on those days? And wouldn't it be useful to store previous computations with different fort/access statuses, as these might be expected to revert back and forth during the war?

Also, instead of computing paths from Africa to Asia on every iteration, would it not make sense to cordon off "regions of operations" (determined by where the wars are) and restrict the computations to each region? This would give a quadratic improvement over doing the computation over the entire globe, after all.
We indeed avoid recomputing when we can, but moving armies are just the tip of the iceberg.
When you the player give an order it will carry on and only be rechecked on specific circumstances to prevent exploits (blackflag to avoid ZoC, etc...).
The big cost is AI armies. They need to analyze multiple paths and assess which one to take (to avoid being ambushed by a stronger army on the way) and need to re-assess multiple times because circumstances might change (as a result of another army moving to intercept or a more pressing matter popping up).
All this of course is somewhat oversimplified for the sake of this DD.
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
It's actually quite easy to reproduce without a save. Just make sure your army enter a province while an enemy leaves the province on the same day. But because of your tag id is lower on the country id list than the enemy, means he will move first on the same day. But on the UI will say there will be a battle.
This is why you should play as Sweden :)
Joking aside that's an interesting analysis, I'll forward it to the team.
 

Caligula Caesar

Stellaris Technical Scripter
64 Badges
Nov 1, 2014
700
847
  • Europa Universalis III Complete
  • Europa Universalis IV: Res Publica
  • Europa Universalis IV: Third Rome
  • Heir to the Throne
  • Europa Universalis IV: Call to arms event
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Art of War
  • Divine Wind
  • Europa Universalis III
  • Europa Universalis IV: Pre-order
  • Hearts of Iron IV: Cadet
  • Stellaris: Lithoids
  • Crusader Kings II: Monks and Mystics
  • Hearts of Iron IV: Together for Victory
  • Stellaris: Leviathans Story Pack
  • Stellaris: Digital Anniversary Edition
  • Europa Universalis IV: Rights of Man
  • Europa Universalis IV: Mandate of Heaven
  • Crusader Kings II: Reapers Due
  • Europa Universalis IV: Common Sense
  • Hearts of Iron IV: Death or Dishonor
  • Europa Universalis IV: Dharma
  • Europa Universalis IV: Cradle of Civilization
  • Stellaris: Distant Stars
  • Hearts of Iron IV: Expansion Pass
  • Stellaris: Humanoids Species Pack
  • Stellaris: Apocalypse
  • Europa Universalis IV: Rule Britannia
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Crusader Kings III
  • Stellaris: Federations
  • Age of Wonders: Planetfall
  • Imperator: Rome Deluxe Edition
  • Crusader Kings II: Holy Fury
  • Crusader Kings III: Royal Edition
  • Europa Universalis IV
  • Europa Universalis III Complete
  • Europa Universalis III Complete
  • Cities: Skylines
  • Europa Universalis IV: El Dorado
  • Crusader Kings II: Way of Life
  • Europa Universalis IV: Golden Century
  • Crusader Kings II: Horse Lords
  • Europa Universalis IV: Cossacks
  • Crusader Kings II: Conclave
  • Stellaris
  • Crusader Kings II: The Republic
  • Crusader Kings II: Rajas of India

StarNaN

EUIV Producer
16 Badges
Aug 17, 2013
950
279
  • Europa Universalis IV
  • Victoria 2
  • Warlock: Master of the Arcane
  • Europa Universalis IV: El Dorado
  • Europa Universalis IV: Common Sense
It's actually quite easy to reproduce without a save. Just make sure your army enter a province while an enemy leaves the province on the same day. But because of your tag id is lower on the country id list than the enemy, means he will move first on the same day. But on the UI will say there will be a battle.

Did a quick search and I couldn't find a thread on bug reports forum.

To give some background information, we fixed a bug on this for 1.17. The UI used to always say there was gonna be a battle, even if there wasn't. This was fixed so that the UI would use the same calculations as the game whether there would be a battle or not.
If I remember/understood it correctly, it used to decide if there should be a battle or not based on the tags. Not the index of the tag, but alphabetical order.

After initial testing on it now I could at first not reproduce your bug, but after some additional testing I found something that might be plausible (but I don't fully know until a programmer confirms it):
The game decides if there should be a battle or not, based on index of tags
The UI says if there should be a battle or not, based on the alphabetical order of tags

If this is the case, something either changed or broke since that patch (or it never worked properly).

What I am trying to say, if you ever have bugs, even if they are super obvious to you, please please please make a thread on the bug reports forum and attach saves.
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
Maybe this has been answered already but I couldn't find anything, so... Is something being done about the "Loading history files" part of the startup taking forever? Every other step of starting the game is fairly quick, but the history files part always takes one or two minutes for me. My computer isn't exactly new but it's not ancient either; 4th gen i5 CPU, 16gb ddr3 RAM and an SSD disk.
I take you're playing on Windows? NTFS (the way Windows arrange your files on your disk) is notably bad at accessing a lot of small files (such as our history files).
This is the main reason why Linux and MacOS starts much faster.
An obvious solution would be to put all core data in one big zip file but it's trickier than it sounds.
Will it help to find "memory leak" problems (or whatever)? I mean problems starting after playing several hours? Like spending some time in modded trade view, selecting provinces in peace deal and then oops... you try to save game, but no existing files shown and trying to save lead to crash without saving of course. And some other crashes on save.
I don't recall a problem like this. Did you make a bug report?
Today's crash dumps are already pretty telling on what went wrong.
The tool itself will help for those edge cases where your game crashes but we can't find a reproducible way to trigger it.
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
I tried to most accurately represent the screenshot, here are the differences I found:
  1. Borders are cleaner
  2. Stripes on occupied provinces or in this case, provinces in the HRE owned by non-HRE countries, stripes continue pas the border and doesn't "fade out", instead it continues until the province limits.
  3. Image of country shields at the bottom doesn't "cut" their shadow like they do now: View attachment 470184
  4. It seems like the yellow stripes, orange stripes and elector colours are brighter/lighter
  5. You can zoom in further (if I zoom in further right now it angles the map kind off horizontally)
  6. Bigger space between border lines (perhaps they seem smaller because of low quality borders, dunno)
  7. EDIT: Province borders are thinner and cleaner/sharper as well as darker, I think state borders are also darker and sharper
If someone has found anything else, please share!
View attachment 470176
I might have been a bit confusing here.
When I said "play 7 differences" it's a reference to a french kid game that may or may not translate well.
In this particular image there should be mostly one difference, but maybe you'll spot something I didn't :)
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
@MatRopert Will these GPU changes fix the odd coloring issues that are seen every now and then? Example: Savoy's pink colour tends to be very grainy, not 100% the same colour.

Or province coloring not covering all of a border province.
I'm not sure... A screenshot in a post on the bug report board (with save and possibly dxdiag output) would certainly help us give a proper answer ;)
 

StarNaN

EUIV Producer
16 Badges
Aug 17, 2013
950
279
  • Europa Universalis IV
  • Victoria 2
  • Warlock: Master of the Arcane
  • Europa Universalis IV: El Dorado
  • Europa Universalis IV: Common Sense
I knew about this bug, and I did ask about its status in Quick Questions, Quick Answers quite recently. I though it was well known, many people answered so it kept me in my assumption. I simply thought that for some reason it was decided that it was not to be fixed.

I remember I had so much butthurt when I was playing Mzab (a later tag) against Portugal and Spain (early ones, so higher in the list?), trying to outmanoeuvre them outnumbered in North-West mountains of Africa, which required me to try to squeeze my armies within particular dates, without battle happening, and quite often battles happened without Battle Indicator, and vice versa, but I can't provide any save game now.

Would a save after the battle happen be enough, or you would have to have a save from before that battle? I usually play Ironman, so that would be hard(er) to achieve.

Usually, saves before or during an issue is happening helps way more than a save after it has already happened. Understandably, it's very hard to have saves from before an issue happens and saves after is better than nothing. For this particular issue, I now have a save from when it is happening. :)
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
Let's talk about the objectively most important question raised by this dev diary:

Romanée-Conti vs Pomerol intensifies...
The huge question is "Are Bordeaux winemakers right in mixing grapes differently every year depending on harvest to get a nice blend or are they alchemists who have forsaken the art?"
 

MatRopert

HoI4 Tech Lead
2 Badges
May 11, 2018
254
741
mropert.github.io
  • Battle for Bosporus
However I think it is a valid observeration/assertion that, crudely, more tags = more lag.
It's interesting that you bring that up as lag.
At the risk of being pedantic, it's best to distinguish some things in that matter:
  • Lag is tied to latency (ping) and usually due to either slow network connection, the server being overloaded or your machine being too slow to keep up with the host speed. Symptoms are usually a delay between a player input and an actual effect in the game. It's most commonly seen in shooters when you shoot some dude in the face, but instead of the expected result (said dude dying of bullet wound to the face), your target moves and your bullet hits whatever was behind him. Basically what your game shows is a couple in-game time units behind what it actually is. It is tied to the nature of multiplayer games and will not happen in singleplayer. Ideally you should play at a speed in which all clients can keep up with the host without ever needing to slow down or pause to allow them to catch up.
  • Stuttering (french: "ça rame grave") happens when your computer is too slow to render the game graphics. It is usually synonymous with frame drops, like rendering a video with a low FPS count. It is normally due to your GPU being too slow for the game (or we did something terribly wrong :) ).
  • EU4 employs some tricks to keep your framerate stable when you CPU can't catch up with the game speed, your graphics might still look fine but the daily ticks pass slowly and your actions take some time to be processed. This is an issue tied to the many computations done every ingame day. It is more noticeable later in the game. Pausing the game should make the problem disappear and allow you to give as many orders as you wish as fast as you can.
Lag is not something we can do much about, but the others we try to investigate.
The most important ones are when the game stutters, even paused. This shouldn't happen since the rendering of EU4 is cheap by modern standards. Sometimes we miss something (like a recent trade company alert performance bug) but it should always be fixable.
Daily tick getting slower is trickier because it's tied to the complexity I mentioned in the original post. As I mentioned we do make sure it stays below a reasonable average every patch. The biggest reason for this is AI, and the fact that almost every day each country needs to make decisions, and there are a lot of them and a lot of things they can potentially do. For fun you can use the 'ai' command in the console to turn it off (in SP non-ironman) and you should see a drastic difference in speed.

Hope this clears things up and helps you describe your issue(s) better.