Bonjour everyone! It is I, Le French Paradox and I will be your host for this Dev Diary.
Today we will talk about Tech Debt. As @DDRJake has been saying, tech debt has been an important focus for the EU4 team since the release of 1.28.3, the latest support patch we did for the Spain update. While our designers are thinking very hard about the yet-to-be-announced big expansion that is to come, we programmers have been busy tidying up the EU4 codebase.
Microsoft should team up with Netflix, I would watch the hell out of this show
Jokes aside, we keep talking about Tech Debt but realize it might be a bit confusing, especially for our non-programming crowd.
While EU4 was announced at GamesCon 2012, the actual development started a year before, which makes the code 8 years old. And of course, the team at the time didn't reinvent everything, bits were pulled from EU3, some of which are still in use today. I'll let you speculate which is which in the comments.
As I'm sure you all know, technology is a rapidly evolving thing, and software engineering is no exception. In 10 years the development practices have evolved, both because the industry keeps learning and inventing new ways to solve problems, and because the ecosystem (the hardware and software that runs our games) also changes.
10 years ago, SSDs were not that widespread, 1080p was pretty new, DirectX 10 was all the rage in previews and most people were so hostile to Windows Vista that they kept running XP in 32 bits.
That is not to say that EU4 was using only stuff from the late 2000s or early 2010s until now. Over time our engineers have done their best to keep the game up to date, but some stuff inevitably slips through the cracks. It's the accumulation of all this aging stuff that we call tech debt(*).
Here's a few examples of those things that we have tackled so far.
64 bits
As the BattlePope already explained in a past DD, 32 bits was a reasonable choice when EU4 development started, but today 64 bits in the norm.
With that done EU4 will keep running with the next update of MacOS (which removes 32 bit compatibility altogether) and also be able to use more RAM. While there are no immediate plans to increase the memory usage of the game significantly, mods that add a lot of provinces and tags should be able to break the previous ceiling.
Rendering
It is no secret that our games do not use cutting edge graphics. True to its board game heritage, Europa Universalis is mostly about showing a world map with some cool dudes (and elephants) fighting on top of it. Yet if you take a look at the Imperator preview, you will see that our graphics experts have learnt a trick or two since EU4's release.
While Boromir is (was?) right, we still managed to replicate some of the improvements they have made. The main one is that the colorizing of the provinces on the map is now done mostly on the GPU, while in the past it used to consume precious CPU cycles to display whatever horrible experiment in bordergore your game was about.
Let's play 7 differences, can you spot what changed from the current release in this beta screenshot?
Crash reporter
Our game never crashes, it just tactically exits. Yet when it happens, we want to know all about it. It helps our QA to reproduce the issue and our programmers to isolate the piece of code responsible and fix it.
Since EU4's release our engine team has made a new and improved crash reporter that brings us more data and help find problems faster. First of all, it now works on Linux and MacOS, meaning we will be able to investigate issues on those platforms much easier. Secondly, it allows us to add some metadata to the dump, such as the current year, the list of enabled mods or how many 6/6/6 heirs died to a hunting accident, helping us understand what triggered the issue.
Startup time
Improving the loading time of the game is something we always want to do. While we can't really apply the best fix to the issue (discontinue Windows support, the game just starts so much faster on Linux and MacOS, trust me), we managed to find some things we could do.
The most notable one was the upgrade of PhysFS, a 3rd party software that a lot of videogames use to load all resources (files on disk, mods, DLCs...). While still far from Unix performance, it should shave off a couple seconds from the startup time on Windows.
Experienced Byzantium players recommend playing on Linux or Mac due to the very fast restart time
General performance
The speed of the game is, of course, always a concern. Every night, we run a couple games on benchmark machines and always make sure that the averages are below a certain value (80ms per in-game day is our current high threshold).
While you may think that performance would improve over time (due to new hardware for example), in practice the average usually go up during the development phase of a new expansion, and is then worked upon to bring it back to acceptable values.
Contrary to popular thought, adding new features in the game is not the only (or even main) source of performance regression. In fact, one big factor is simply the addition of tags and provinces.
Consider this: for every two countries in the game, a bunch of stuff needs to be computed each in-game day (relations, AI attitude...). For every two provinces in the game, the game has to know the way from A to B (what we call pathfinding). This is known as a quadratic problem. For those who don't favor math, it means that for X provinces, the problem complexity is X squared. Doubling the number of provinces does not double the number of computations needed, it quadruples it. Over the years, part of our job has been to allow EU4 to grow from around 2000 provinces to more than 4000 provinces.
Keep in mind though that in software engineering an optimization that makes something 0.1% faster is considered pretty good, 1% excellent and 10% probably means you introduced a serious bug (or fixed a very serious one). How do we manage to improve anything with those numbers? Simple: all those add up in the end.
That's all for today! Should you like to know more meaty details of the tech stuff, you can check-out my blog where I get into more practical details about development practices.
Next week we should return to a more classic Dev Diary by our designers, unless you can help me convince Jake to revamp the wine trade good to include grape kinds and soils and finally settle which of the Burgundian or Bordelaise tradition of winemaking is the best.
(*) It's a bit more complicated than that but should be enough for the sake of this diary
Today we will talk about Tech Debt. As @DDRJake has been saying, tech debt has been an important focus for the EU4 team since the release of 1.28.3, the latest support patch we did for the Spain update. While our designers are thinking very hard about the yet-to-be-announced big expansion that is to come, we programmers have been busy tidying up the EU4 codebase.
Microsoft should team up with Netflix, I would watch the hell out of this show
Jokes aside, we keep talking about Tech Debt but realize it might be a bit confusing, especially for our non-programming crowd.
While EU4 was announced at GamesCon 2012, the actual development started a year before, which makes the code 8 years old. And of course, the team at the time didn't reinvent everything, bits were pulled from EU3, some of which are still in use today. I'll let you speculate which is which in the comments.
As I'm sure you all know, technology is a rapidly evolving thing, and software engineering is no exception. In 10 years the development practices have evolved, both because the industry keeps learning and inventing new ways to solve problems, and because the ecosystem (the hardware and software that runs our games) also changes.
10 years ago, SSDs were not that widespread, 1080p was pretty new, DirectX 10 was all the rage in previews and most people were so hostile to Windows Vista that they kept running XP in 32 bits.
That is not to say that EU4 was using only stuff from the late 2000s or early 2010s until now. Over time our engineers have done their best to keep the game up to date, but some stuff inevitably slips through the cracks. It's the accumulation of all this aging stuff that we call tech debt(*).
Here's a few examples of those things that we have tackled so far.
64 bits
As the BattlePope already explained in a past DD, 32 bits was a reasonable choice when EU4 development started, but today 64 bits in the norm.
With that done EU4 will keep running with the next update of MacOS (which removes 32 bit compatibility altogether) and also be able to use more RAM. While there are no immediate plans to increase the memory usage of the game significantly, mods that add a lot of provinces and tags should be able to break the previous ceiling.
Rendering
It is no secret that our games do not use cutting edge graphics. True to its board game heritage, Europa Universalis is mostly about showing a world map with some cool dudes (and elephants) fighting on top of it. Yet if you take a look at the Imperator preview, you will see that our graphics experts have learnt a trick or two since EU4's release.
While Boromir is (was?) right, we still managed to replicate some of the improvements they have made. The main one is that the colorizing of the provinces on the map is now done mostly on the GPU, while in the past it used to consume precious CPU cycles to display whatever horrible experiment in bordergore your game was about.
Let's play 7 differences, can you spot what changed from the current release in this beta screenshot?
Crash reporter
Our game never crashes, it just tactically exits. Yet when it happens, we want to know all about it. It helps our QA to reproduce the issue and our programmers to isolate the piece of code responsible and fix it.
Since EU4's release our engine team has made a new and improved crash reporter that brings us more data and help find problems faster. First of all, it now works on Linux and MacOS, meaning we will be able to investigate issues on those platforms much easier. Secondly, it allows us to add some metadata to the dump, such as the current year, the list of enabled mods or how many 6/6/6 heirs died to a hunting accident, helping us understand what triggered the issue.
Startup time
Improving the loading time of the game is something we always want to do. While we can't really apply the best fix to the issue (discontinue Windows support, the game just starts so much faster on Linux and MacOS, trust me), we managed to find some things we could do.
The most notable one was the upgrade of PhysFS, a 3rd party software that a lot of videogames use to load all resources (files on disk, mods, DLCs...). While still far from Unix performance, it should shave off a couple seconds from the startup time on Windows.
Experienced Byzantium players recommend playing on Linux or Mac due to the very fast restart time
General performance
The speed of the game is, of course, always a concern. Every night, we run a couple games on benchmark machines and always make sure that the averages are below a certain value (80ms per in-game day is our current high threshold).
While you may think that performance would improve over time (due to new hardware for example), in practice the average usually go up during the development phase of a new expansion, and is then worked upon to bring it back to acceptable values.
Contrary to popular thought, adding new features in the game is not the only (or even main) source of performance regression. In fact, one big factor is simply the addition of tags and provinces.
Consider this: for every two countries in the game, a bunch of stuff needs to be computed each in-game day (relations, AI attitude...). For every two provinces in the game, the game has to know the way from A to B (what we call pathfinding). This is known as a quadratic problem. For those who don't favor math, it means that for X provinces, the problem complexity is X squared. Doubling the number of provinces does not double the number of computations needed, it quadruples it. Over the years, part of our job has been to allow EU4 to grow from around 2000 provinces to more than 4000 provinces.
Keep in mind though that in software engineering an optimization that makes something 0.1% faster is considered pretty good, 1% excellent and 10% probably means you introduced a serious bug (or fixed a very serious one). How do we manage to improve anything with those numbers? Simple: all those add up in the end.
That's all for today! Should you like to know more meaty details of the tech stuff, you can check-out my blog where I get into more practical details about development practices.
Next week we should return to a more classic Dev Diary by our designers, unless you can help me convince Jake to revamp the wine trade good to include grape kinds and soils and finally settle which of the Burgundian or Bordelaise tradition of winemaking is the best.
(*) It's a bit more complicated than that but should be enough for the sake of this diary
