HOI4 Dev Diary - Our development process

[KimchiViking]

@podcat asked me to prepare a Dev Diary from a Project Lead perspective...
Quick background on me: I came on as Project Lead for HOI4 shortly after the game was released, last summer. That’s my perspective. I speak from a expansion-development point of view, and our work going forward.

The purpose of today’s Dev Diary is to give you a little insight into our development process.
Why we fix some bugs and leave others... How we decide what goes into the next expansion... and more.


Imagine, if you will, a crowded bar
Imagine all the customers screaming out their orders at once, to a single hard-working bartender.

How many words do you think the bartender will be able to make out, over the collected noise of the crowd?
How many drink orders do you think he will he be able to get right?

If you answered “none”, you’d probably be close to the truth.

My job, if I’d been working in that bar, would be to to organize a queue-system that works.
To move things along and make sure that people get what they want, as quickly as possible.

As project lead at Paradox; it’s my job to make sure that our players get what they want.
Best possible value in the game, within the shortest possible amount of time.

Here is a super condensed version of how the team and I go about making this happen.


The Design Process
First, within the team; the designer speaks for the players. They’re the one that decide what the team should do.
(Game director @podcat and game designer @Pallidum , in our case.
Continuing with the bar analogy; they’re the people who have a feel for the market and decide what goes on the drinks menu.)

New content to keep players engaged and happy... Weaknesses in the game that needs to be addressed... A balanced mix of all that good stuff is collected together in a “design document”.

The document explains the vision for each new feature, or fix, that goes with a new expansion. It essentially serves as a specification, or commission for work to be done, for the team.


How do we decide what new features and fixes go into an expansion?
The designers base their decisions on what goes into the design document, on, for example:

• Do the features fit into the overall theme of the expansion?
  (This also goes for bugfixes where we prefer to work by theme. For example Air or Naval).
• Do we hit a good mix of paid features and free features?
  (A lot of this is decided on how difficult things are to implement and their impact on the game’s balance.)
• Data we collect on player behavior.
  That data is analyzed and lead to new features or fixes.
• We have a database full of suggested improvements.
• Not to mention bugs that we prioritize and work off, in priority order.
• We also closely monitor mainly this forum, and (to a lesser degree) other HOI-communities, in case something pops up. Both bugs or inspired posts in the suggestion forum.

(For the love of God, YES! We saw the forum bug report.)


How do we choose which bugs to fix? (A bug’s journey from the bug forum to being fixed)
As I mentioned; we have a big database with bugs, improvement ideas and feature-suggestions.
(Really big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist, but that's just peanuts to our database.)

A lot of entries in this database, however, are related to the same underlying system. Doing an overhaul on that system will get rid of a whole bunch of bugs. These are things we prioritize.

A bug often starts out coming onto our radar by being reported, here, in our bug forum.
(I really want to stress this point, because we occasionally see people posting bug reports on reddit or other places. The odds of someone from Paradox stumbling over those reports and carrying them forward into our database are slim.)

The bug is transferred from here into our database. And we start looking into it, by analyzing it.
We need to know how frequent the problem is. How serious, and how quick it is to fix.
The more frequent or serious it is increases its chances of getting fixed. Soon.
If it also happens to be quick to fix… well, that’s just a win-win.

If a bug is serious, frequent and quick to fix and it’s still not being fixed… The most likely explanation to why we’re not fixing it ; is because we simply couldn’t fit it into our schedule.

It might help to understand this, if you…

Think of the development process as a single work day...
Serious things you hear about, before lunch, will get fixed before the day is done. For sure.
Then you might work on something else, with lower priority, for a while.
Until the next big problem pops up.
But, by then, you can’t start on it. Because you can’t get it all the way done, before you have to go home. It’ll have to wait until the next day.
So, in order to not waste precious time, you squeeze in something else that will fit.

This is how our development cycles work. Sometimes we simply can’t start on something and get it fixed, or improved, before the expansion has to ship.
(This also illustrates how sometimes things with lower priority get done when some higher prio stuff are left for later.)


Difficult judgement calls
Other bugs or suggestions are more up for debate.
Doing something that will make one group of people leap for joy - might seriously anger another group. We have to stay on top of that.


The big time-stealers
Not to mention that some requests, like improving AI, is a perpetual job that can’t be rushed.

As obvious as it is that an area needs work; some things are like hatching an egg. It takes the time it takes. No matter how many bodies you throw on the problem.

(Btw, this is how I imagine a Steam Summer Sale going. If Steam was a physical store.)


The Breakdown & Estimation process
Moving on: Once the design doc is complete… The team takes the design and breaks it down into bite-size tasks for coders, content designers, art, UX-design, sound etc.

When we have everything broken down into a list of tasks; we sit down together and “estimate” each task. Giving us an idea of how long the full feature will take to develop, once you add it all together.


How are deadlines and release-dates determined?
Paradox has a plan for how many expansions/DLCs we should release per year.

HOI4 release dates are determined based on: 1. that plan, 2. how quickly we can reach the desired sell-value of the release, and lastly 3. coordinated with specific dates that our marketing team have selected.
(More on this subject in next week’s Dev Diary.)


Can we make the expansion-design happen within the deadline?
After all features have been estimated; I can figure out if what we want to do is possible within the deadline. With the people at our disposal.

If yes: Huzzah!

If not: This is where I have to crush the designer’s hopes and dreams.

Splat!

We need to cut something in order to be able to finish on time.
This is something we discuss and agree on, together. While I gently pat their backs and hand them tissues.


What gets cut?
When cutting something I have to consider, for example:

• The desirability and priority of the feature.
• What people we have available.
  • How much, and what, each person can work on.
  • While not being blocked, or blocking, someone else.
• What features tie into other features.
  (If there is anything independent enough to cut cleanly.)

Sometimes laying this complex puzzle, trying to fit high priority pieces together, is trickier than trying to nail jello to a wall. Things slip and change constantly. This is the very essence and nature of development work.

In closing
Speaking of the nature of development work… While the example above mostly serves illustrate problems with communication, which is always a factor when people with different perspectives discuss something… I think it says something about how frequent certain development problems are; that a site exists where you can create your own project cartoon, like this one.

The issues that Paradox and HOI4 struggle with are the same problems that all IT projects, everywhere, grind their teeth over. It’s terribly complex work. Which is why, although the problems and risks are well known and can be anticipated and planned around, to a degree… they remain.

The silver lining, I think, is that while our problems are the same… we at Paradox have a hell of a lot of fun while working on them.

Next week, our Brand Manager will write a Dev Diary. Before handing the baton back to podcat.

Don't forget to tune into World War Wednesday today at 16:00CEST on https://www.twitch.tv/paradoxinteractive! To see podcat run Germany in massive co-op, with the other devs as generals.

 

[podcat]

I personally find things like this quite interesting. Not working in development myself, it's nice to hear about what goes on behind the scenes.

I prefer not to have more breaks than we have to with diaries. I get that ppl preferably would want to get details on features etc, but rather than a break I hope people will find some *actual* development diaries interesting to read too.

 

[KimchiViking]

'You can please some of the people all of the time, you can please all of the people some of the time, but you can’t please all of the people all of the time'

In so many words. Yes.

 

[KimchiViking]

I find this interesting, and a good read. It's always nice to know the process you guys go through. But one thing that needs improved in this is the justification for the costs of your DLC's. The general reaction to Death and Dishonor wasn't the best- due to the price for such a small amount of content.

If you have questions on this subject; I suggest you note them down for next week. That's when our Brand manager will write a DD and be available to answer questions.
(Pricing is not mine, or the rest of the dev team's, area of expertise.)

 

[KimchiViking]

This was a very nice diary! I'd love for these to crop up regularly, and ideally from your other development teams too
Thanks!

Glad to hear it! Unfortunately (and as expected by us) I think you're in a minority.
Most people want to hear about nothing but new cool stuff or fixes, rather than the actual development process. (Heaven forbid there is actual development talk in a development diary, right?)

As podcat said last week; those people will simply have to be patient.
We're starting on all of our big changes first. Which we can't show you yet, as they are time consuming to develop.
As @Gamer_1745 pointed out earler... it was either this or nothing.

 

[KimchiViking]

Yeah I guess developpers don't realize that. They base their judgements on the few dozens of people who always comment on the forums and would say nice things regardless of what is being thrown at them. They don't get the bigger picture, that is, most HOI4 players aren't happy.

People who have something negative to say are far more likely to post than those who have nothing negative to say.

We wouldn't be doing our jobs if we weren't aware of these things.
The forums represent a small and vocal minority.

What I was stressing was the importance that bug reports are posted here.
If they're posted here they get on our radar and has a chance to get fixed.
If they're posted somewhere else or nowhere at all; those odds drop to zero.

 

[podcat]

Maybe you see a different reality. Steam Ratings are in the positive by a huge margin. And the player base looks quite stable to me .... http://steamcharts.com/app/394360 . Looks more like that a lot of people who enjoyed the game from the start still enjoy it on a regular basis.

yeah, when I checked after summer HOI4 was the most played (monthly players) of our strategy titles (cant touch that Skylines ). EU4 is currently beating us with like 1k players tho :'( it goes up and down a bit.

 

[KimchiViking]

What amazes me is that you still have a job after the mess that was year 1, not to mention the reception and reviews of the ''expansions''. Hell, you could write a guide on how to keep your job despite being terrible at it, sell it and become rich.

I must confess... I was wondering how long it would take for someone to write something similar. Page 3. Check.

With the reception of the expansions I have been apart of... there could be a valid question hidden there. Somewhere, under the uncalled for rudeness.

The circumstances that I and the team have worked under for the last year, however, is something that I couldn’t talk about - even if someone asked nicely.

Fortunately for me; people who are in a position to actually see what the team has to work with, and what I do with my responsibility areas for the game, are the ones who have a say in my future with the company.

All that a comment like this does - is showcase (for those of us who do know a thing or two) the poster's utter ignorance and lack of insight into how this or any other development project works.

 

[KimchiViking]

An actual development question! Yay!

When you are prioritizing features, how do you evaluate changes to core mechanics?.

This is one of the trickiest parts of what we do. You touch one little thing and it has ripple effects EVERYWHERE. That's true for most software. But with a game as complex as HOI? Nightmare!

What we do is that we analyze the risk and possible impact of each change before we do anything. We have to weigh that risk against how much time we have to work on something and how badly we need something. It's a part of every move we make (or don't make).

On the other hand, this sort of thing isn't a bug, and it doesn't tend to lead to a lot of people crying doom. In fact, there is an excellent chance that the designers don't even consider it a problem. Is there time in the scheduling to consider things like this, or is it basically not on the radar unless it becomes a priority of the designer?

Making sure that new features or fixes don't throw off balance is always a big and planned part of everything we do. Always.
However, as you hint at... there's pretty much no way that our QA, Beta testers or designers can think of, or try, every possible combination. This is why we always allocate time to patch. Once the general public get their hands on something there is almost always something to tweak.

Are suggestions that include detailed math and code more likely to be considered, or is such "back seat game design" just irritating noise?

Heavens no! Math is porn to some people around here. By all means include it.
You never know what might insire someone who reads it.

 

[KimchiViking]

Well, THAT'S intriguing! Probably a general lack of resources, but I prefer to think that it's fighting off a Nazi invasion through the ill-conceived time portal.

Ssshhh! The time police sees and hears everything. I can't talk about it.

 

[podcat]

When you are prioritizing features, how do you evaluate changes to core mechanics? For instance, I think the way HoI4 treats combat width makes little sense, and distorts division design a lot.

KimchiViking already touched on it but I'd figure I'd say a few more words on this. Core mechanic changes are like this:
+ Makes the game feel fresh and makes you have to learn new strategies etc
- May require a lot of balance work

Because of the balance work we need to make sure we dont do too many at once so we have time to actually balance them. Division design is an area we have a lot of thoughts and discussions about, but its also incredibly large impact on both AI and game balance so basically needs to be handled like a large feature part of an expansion. I think last weeks diary on future plans included a bulletpoint on division designs and making them less gamey (and a bit more historical) and more interesting gameplay wise

 

[podcat]

but may you please refrain from using gifs extensively, please? It's just uncomfortable to read.

Haha, you should see our team presentations

 

[KimchiViking]

Nice diary! Interesting to hear from this point of view. May I ask what kind of education is needed to get such a job? Is it more like a project management or software development (coding) or a mix? If that question is too private to answer, I understand.

Not at all.

My background is as a coder/project manager for 20 years. In IT and software development.
I have accumulated certifications for a number of the industry standard methodologies, such as scrum, kanban and other agile certifications over the years.
So I've done coding. School. I have years of relevant experience. The whole nine yards. (Not to mention being a life long gamer. And long-time strategy- and Paradox fan.)

Mine is not the only way to go however.

We have project leads working here who have started in junior positions and worked their way up. (Once they've proven to have organizational skills, leadership talent and an ability to get things done.)
Everyone has a mix of something, to compliment their management skills. Be it technical, economics or something else.
For new hires, these days, we favor experience in leading complex development projects in IT and games industry.

Best advice I can give someone who wants to get into the business, in one capacity on another, is to work as a coder, QA, project manager, designer in a similar field. And do mods in your spare time. Or create your own game.

Especially if you haven't got previous experience in the games business. This is a great way to stand out and beef up your CV!
We have so many people here at Paradox, and on the HOI-team, who were modders before they joined. They beat out the competition by not only telling Paradox that they were passionate about our games... but they had proof to back that up.

Hope that helps.

 

[KimchiViking]

@KimchiVikingDo you guys use an Agile methodology? If so what tools?

We do, yes. All Paradox games run some form of Scrum. As do we.

Software tools or agile tools?
Most important software tool: JIRA
Agile tools: All the usual cornerstones like daily standups, estimation, sprint planning, retrospectives etc

Your DB sounds suspiciously like a Backlog.

You guessed it. We use JIRA to keep track of it all.

Do you agree dates by a projection of a burn rate versus the point value? Or is it more traditional approach with Gantt charts and RAID logs?

When I first started at Paradox I used Gannt charts, as the project was more waterfall than agile back then.

Now that I’ve been with Paradox a while I’ve been able to transition into using story points to calculate our velocity and keep track via burndown charts.
You can use either. It’s just that I personally prefer SP and burndowns.

Another vitally important thing for us is resource management. For that I use other tools.

 

[KimchiViking]

Just wanted to say I thoroughly enjoyed the diary (the gifs too), thanks!

I lured out a lurker. Yay!

Also; see @podcat ? People like the gifs. The more you mock them then more I'll use them. Resistance is useless.

 

[Archangel85]

I would rather have throwback dev diaries concerning HOI3 and the mechanics to the Sino-Japanese war and HOW you will translate those mechanics into HOI4. We don't need any sort of concrete images or definitive features. What we want is a glimpse into the CURRENT DEVELOPMENT progress. What is the point of making a dev diary if you are just writing about content that you finished a week ago and are already working on the next little blurb. This way maybe players can give some sort of feedback and get information about the game's general direction.

If we did this it would just be a repeat of the Romania dev diary, i.e. a complete meltdown.

I KNOW for a fact that this DLC is concerning the pacific theater which would contain the two Chinas, Manchukuo, Philippines, Siam, and Japan. Naval refits and naval tactics (similar to land tactics cards) will be present as well as an actually FUNCTIONING pearl harbor event chain that could provoke an early war with Japan at the cost of losing a sizable amount of capital ships.

Wow, I didn't know this and I work here! Thanks for letting us know!

 

[Bratyn]

If you''re hiring let me know haha

https://career.paradoxplaza.com/jobs

 

[podcat]

Nice work . EU4's pretty mature now - have you tried comparing players at the same time since development (so release month aligned data) - you should definitely do this if Johan gives out cake to the team that gets the most players or something like that *.

* Obviously, the number of players is not a sensible measure of how good a game is (or even how financially successful it is), I'm just being silly here .

Generally we do cakes by team once we hit certain milestones :3

Could you explain a bit how you define "balance" in terms of this game and how it's getting tested? In a game with singleplayer/multiplayer, where every country has different starting positions, many possible strategies and different (ahistorical) goals...

We dont really balance from a MP perspective but focus on SP, especially the more historical paths. Although we are slowly switching that around to care more about the alt history path difficulty etc. In general there are different balance situations we look at:
1) QA chimes in on if mechanics feel worth it, if stuff isnt worth using etc or if they are overpowered as part of each sprint, we reserve time for dealing with this as iteration time for this on each sprint. They also bring up strategies you can use to cheese ai etc (of course this stuff never ends)
2) Making sure the game follows historical flow on all ai with historical ai on. For that we run 20-30 games and compare statistics. When stuff breaks we take a look as to why. If the reason for the outcome being wierd is a mistake by AI (say germany is losing vs soviets, is it because it made a tactical messup, because it bled too much vs the french or because its production is not up to it) we look at fixing that. If there is no mistake we look at overall balance and adjust factory counts, starting techs, division template focus and the like.

A lot of good balance feedback also come from betas who play a lot of MP games of the kind we dont really at the office (very slow, very heavy use of manual controls etc). Although we always need to take the AI into consideration when balancing too as tailoring for that playstile will make things harder for it.

We also have certain vaguer design goals for balance that we slowly move towards, but require larger changes. For example making early wars less attractive for Germany which comes down to speed of growth vs other powers etc

That last sentence sounds like the speed to fix something is secondary? I.e. let's say bug A is 12x more frequent/serious than bug B but takes 50x longer to fix. Which bug would you prioritize? So far when I reported bugs, I prioritized things that I think are clear-cut and easy to fix.

We specially mark "quick fixes" so they are easier to find and fix. When I prioritize I generally bump easy to fix stuff one level up. As for priority its a tradeoff between frequency and how how much it blocks players. An animation bug that happens every game may still get less priority than fixing a war merge bug that breaks the players WW2. MP and non-windows platforms tend to get lower priority in general as they affect a lot less people etc. We try to aim for bang for buck (also to go with the patch theme as we found it to be a lot easier to focus heavily on one area for testing and fixing rather than just spread stuff around priority)

I am inherently suspicious of any design process that didn't include getting drunk at some point but always curious to see the inner workings of a creative process for a studio like this. Made for a neat read even if nothing didn't jump out as a surprise.

Most of the serious design work is done with some form of alcohol involved Brainstorming->detail design matches roughly: whisky->wine->coffee scale

fix the event for germany getting their half of poland from russia using the MR pact if russia eats poland instead this patch

Correct me if I''m wrong, but that only happens if they sign the pact right? why wouldnt both sides honor the pact? or am I getting this wrong

Lastly, is your JIRA linked up to a code management tool? (e.g. GitHub)

Only softly (ppl write revisions and such in jira). We have been looking into more integration with https://www.atlassian.com/software/fisheye but the judgement is still out on that one a bit.

 

[Bratyn]

^ I approve of the book. Reading it myself right now.

 

[Bratyn]

Which one of them, the open scribbly one or the one by Churchill?

Churchill.