CK2 Dev Diary #57: A Bug's Life

[Meneth]

Good afternoon, all. I’m Magne “Meneth” Skjæran, the programmer on CK2. In the past I’ve written dev diaries about modding, optimization, and quality of life improvements, as well as last week’s dev diary talking about what each role on the team is responsible for.

Summer is still ongoing, and most of the office is gone until the end of July, including the majority of the CK2 team. The show must still go on, so for these four weeks I’ll be writing most or all the CK2 dev diaries.

Since most of the team being on vacation means that little development is happening, that means that these four dev diaries will be pretty light on info about the upcoming expansion and patch, but I hope to provide you with some interesting insights, info, and tidbits regardless.

Today, I’ll be talking about the life of a bug; essentially a view into how a bug goes from discovery to release of a fix, and who is involved at each stage.

At the end of this diary, I’ll also share five changelog entries from the 2.8 patch chosen (mostly) at random.

The Bug
To illustrate the process, I’ll be using the following bug as a case study of sorts:
- Fixed it in some cases being possible for priests of religions where priests are not allowed to marry to marry or betroth someone

This bug fix was included in the 2.7.1 patch. What it essentially did, was fix the Pope ending up with a family member that loves Divine Blood marriages so much, they’d betroth the Pope, even though he’s not allowed to marry. Xwedodah is a powerful thing, but this would always end with tears: the moment the Pope’s niece came of age, the betrothal would get broken.

In retrospect, this changelog entry should clearly have been “Fixed the Pope loving Xwedodah so much that he’d get betrothed despite not being allowed to marry”. You live and learn.

Discovery
This bug was discovered by one of our beta testers on 12th February, 2017 while playing the game; they noticed that the Pope would get matrilineally betrothed to his niece, and had a save from before this where it would consistently happen.

Discovery by beta testers is however only one of many avenues of discovery. Other ways include:

Playtesting: Quality Assurance testers spend some of their time simply playing the game, though usually with some kind of focus based on what we’re working on. For example, during Monks and Mystics development the focus might have been participating in a specific society. During playtesting, they’ll report any bugs they run into, as well as making note of balance issues, usability issues, and similar. Depending on the stage of development, these will also be reported directly; if the system is a work in progress, the developer responsible might simply be poked with the issue, but if it is supposed to be finished, a bug report is generally made.

Task verification: When a task (for example, “add a prisoner mass action system”) has been completed, quality assurance will verify that it works according to specification. If they find issues during this time they will either reopen the task, or make bug reports about the task. You can read more about verification in the “Verification” section of this DD.

Non-QA developers: The non-QA devs play the game as well, and are expected to report any bugs they encounter, even though finding bugs is not their primary purpose on the project.

Forum monitoring: While the other bug discovery methods are focused on discovering issues before the patch is released at all, things do slip through the cracks. When they do, they’re often discovered by regular players of the game, some of which end up making bug reports on the forums, on reddit, or similar. Quality assurance spends some of their time monitoring these places, making internal bug reports when issues can be replicated. If the bug cannot be replicated based on the information provided, they will usually ask for more information. An internal bug report will only be made if either the issue can be replicated, or a lot of people are having it. So if you want to make sure your bug report has the best possible chance to result in a fixed bug, you should always provide reproduction steps and a save if at all possible!

Bugs are found in a number of other ways as well, but these four avenues and beta testing cover the majority of cases.

Reporting
Once a bug has been found, it has to be reported internally. Our bug database is contained in the project management web application Jira, which provides a variety of ways to easily manage issues so that they can be scheduled appropriately and things don’t slip through the cracks.

Above you can see the report for the Pope betrothal issue.

The main things needed when reporting are:

Title: A title summarizing the issue. This is supposed to follow the format: Project - Expansion - Area affected - Description of the issue. In this case, “CK2 - MnM - AI - Pope gets engaged matrilineally to the daughter of his kin king”. The title should make it clear at a glance what the issue is and when it was discovered.

Description: The description includes any details that can’t be fit into the title. This often includes a description of what the player was doing when running into the bug, theories about the cause, or in the case of bugs discovered via forum monitoring, the source of the bug report.

Steps to reproduce: In some ways the most important section, the steps to reproduce says how the issue can be replicated. Without this, most bugs are difficult to impossible to fix, as just the information that something can happen often isn’t enough to figure out how it can happen. This is used by both the person fixing the issue (see “Fixing”) and the person verifying the fix (see “Verification”).

Result: Reports also include both the expected result and actual result, to make it even clearer what the actual issue is. Often it is relatively obvious from the other fields, but especially for things like balance issues, these sections can be of great help. They also make it easier to figure out if a problem has actually been fixed or not (see “Verification”).

Attachments: Often a screenshot of the issue is attached. A save is usually included as well if that makes reproduction easier. For crashes and similar, logs are also included, as well as a crash dump which can be used by the programmer to see where in the code the game crashed.

There’s a number of fields beyond this, but they’re generally set later in the process.

Assignment
Once a bug has been reported, it has to be decided who is responsible for fixing it, how important it actually is to fix, and when it should be fixed. Responsibility is first done by discipline; a bug will be assigned to Script, Code, or Art, depending on what causes it. This assignment is usually done by the quality assurance testers if it was not set when the bug report was created.

Priority is set depending on how important a bug is to fix, and ranges from 0 (Must Fix Immediately) to 5 (No Priority). A crash for example would be priority 1 - Must Fix, while the Pope betrothal bug was simply priority 3 - Medium. The project lead has the final call on priority, but generally quality assurance will set the priority for bugs, while the project lead prioritises tasks.

Similarly, there’s Severity, which ranges from Blocker to None. Rather than being how important fixing the issue is, this represents how broken it is. The two correlate to some extent, but you could have something like a highly visible spelling error being High Priority while being only Minor Severity. Severity is set by QA rather than the project lead.

Then, it has to be decided when the bug should actually be fixed. This is done by assignment to a sprint (a period of time during which a specific set of tasks are to be completed. On CK2, these last 4 weeks). For important bugs, they’re generally assigned to the sprint currently ongoing, while less important ones are generally left for the polish weeks rather than assigned to a regular sprint. The exception is bugs relating to new features; those are nearly always assigned to the ongoing sprint even if they aren’t especially severe. This is because it is considered more important to ensure new bugs aren’t added than to fix bugs that are already in the released product. However, the older bugs still do get assigned to the polish weeks, once the main work on the expansion itself is done. During feature sprints, the last of four weeks is dedicated to fixing bugs; during the first three weeks generally only bugs that block other work will get fixed.

When the sprint with the bug comes along, the bug is then assigned to a specific person. The project lead has responsibility for this, but it is often delegated to the discipline itself. For example, the content designers generally divide bugs between themselves autonomously rather than having the project lead assign them.

Fixing
All issues in the sprint more important than the bug have been resolved, so the time has finally come to fix it. The person assigned the bug will now hit “Start Progress” on the bug, thus informing the rest of the team of what they’re working on.

The person will then go through the steps to reproduce, and try to figure out where things are going wrong. In the case of the Pope betrothal, the bug was assigned to me, and the steps to reproduce worked flawlessly. This allowed me to set a “breakpoint” in the code for the AI marriage logic, pausing the code execution when someone would betroth the Pope. This allowed me to work upwards until I found where in the code the theocratic requirement on marriage was located. In the end, it turned out the “CanMarry” function never actually checked for this, only the marriage interaction itself did. As such, betrothals could still happen, but would get invalidated when the characters involved came of age and tried to marry.

End result was one new line of code in the CCharacter::CanMarry function:

if ( ( GetGovernment()->IsTheocracy() && !GetReligion()->CanPriestsMarry() ) || ( pCharacter->GetGovernment()->IsTheocracy() && !pCharacter->GetReligion()->CanPriestsMarry() ) )
        return false;

This simply checks if either character is a theocrat belonging to a religion that doesn’t allow priests to marry.

After recompiling the game, I then ran the steps to reproduce from the bug report once more, and saw that the betrothal no longer happened.

Once the fix itself has been made, a changelog entry is written. The fix and changelog entry are then committed to our repository for the game. Our build servers then compile the game, and a few hours after midnight, a new version of the game is uploaded to Steam for our beta testers to play.

The bug is then marked as resolved as “Fixed” with a message summarizing what was done, and at what revision the change was made. In the case of the Pope betrothal, it was marked as fixed 6 days after it was reported, but this can easily vary from minutes to months.

Verification
Next up, quality assurance verifies that the bug is actually fixed, and that new issues in the area haven’t appeared as a result.

This is done by following the reproduction steps to see if the issue still occurs, and if the issue is especially complicated, trying to reproduce it in other ways.

As mentioned in the “Discovery” section, quality assurance also does playtesting to discover new bugs, and this often includes bugs that are unintended side-effects of fixes to other bugs.

When QA is satisfied that the bug is indeed gone, they mark the issue as closed, and the fix process is typically over. If the bug actually still persists in some form, the bug is reopened, and we return to the “Fixing” step.

Sometimes a bug will resurface even after verification. This generally leads to the issue being reopened and fixed once more.

In the case of the Pope betrothal, the bug was verified 45 days after it was resolved as fixed. We try to avoid having verification lag more than a sprint or so (a month) behind resolution, as it is generally easier to go back and fix something properly if you worked on it recently.

Release
The bug has been fixed, and it is time for a patch, but the process isn’t quite complete yet.

While the specific bug has been verified as fixed, the project as a whole needs to go through one last step to be considered ready to release: the smoke test.

The smoke test essentially checks that nothing is horrifically broken on the three operating systems we support: Windows, Mac, and Linux. Unlike the other steps in the process, the smoke test is not conducted by the CK2 team, but instead by PDS’ central QA team.

Once central QA has signed off on the release, no more changes can be done. Some days to weeks later, the patch is released, and the bug is now fixed for all players.

Summary
All in all we have a number of stages to the bug fixing process to ensure that bugs are found, fixed, and stay fixed. Despite this, bugs do slip through the cracks as CK2 is a rather complicated beast. There’s also always known bugs in each release, as at some point release has to happen, and fixing one bug can easily cause another. We try to minimize the number of known bugs by having a long period of time dedicated to polishing the game before each release, and by having a shorter period during which no changes are allowed at all except fixes to critical issues. There will always be bugs, but we’re always working hard to avoid introducing them, and reduce the number of them.

I hope you found this look into how we handle bugs at CK2 informative.

Now as promised, here’s five changelog entries from the upcoming (no public ETA at this time) 2.8 patch:

• The AI has learned how to spell, and no longer look for "soceities" missions when they're actually called "societies" missions. With this newfound ability to spell, the AI is better at identifying mission targets
• Fixed using council consideration letting you bypass the need for a favor on your liege
• Added Holy Order for Manicheans
• Guardians can now be assigned from birth (though there are no effects until age 6.)
• AI should no longer pine for a matrilineal marriage if they have been turned off by game rules

That’s all for this week. Next week I’ll be talking about improvements we’ve made to modding in 2.8 so far.

 

[Meneth]

That was extremely interesting, it's nice to know there is a thorough process to all of this. I take it with hotfixes that you just skip a bunch of steps if the game has a really severe bug?

We don't skip any steps for hotfixes, but the amount of time between each step is much smaller since priorities are changed to ensure the fix can be completed, tested, and released quickly.

 

[Meneth]

Yes, there are another five changelings.

Regrettably these two have already been mentioned in another Dev Dairies. I would be more pleased if I can see something that has not been mentioned yet.

Didn't realize they'd been mentioned already. He's two other minor things to make up for it:
- Fixed Satanist merchant republics getting a inverted cross icon with the wrong background color
- Being assigned to a subunit that is participating in a siege no longer stops commanders from being assigned to a flank

 

[Meneth]

I'm surprised that I don't see any Stash/BitBucket integration, is there a way to trace your commits to your tickets?

On CK2 we use SVN. Whenever a commit relates to a Jira ticket, we include the ticket in the commit message.
E.G.,:
- Fixed it in some cases being possible for priests of religions where priests are not allowed to marry to marry or bethroth someone [CKTWO-8388]

 

[Meneth]

Well written diary.

Err..... are just talking ahead of time or do we not get an official 2.7.2 patch release, apart from the Beta of it ?
Did i miss something ?

2.7.2 will be released before 2.8 will, at some point after people return from vacation.

 

[Meneth]

This was very interesting to read, @Meneth! As someone who has just recently worked with these tools in university, it was cool to read how an actual company does it.

But I kind of wonder: one issue we had during our sprints was assigning things to tasks that aren't generally divisible into tasks. One was research (figuring out how to use Cucumber in our case). And one I could imagine to come up for you is optimisation. A lot of work on that discipline was done before the Reaper's Due patch. So I want to ask: how do you divide something vague like "optimisation" into tasks? Is it like, everybody gets a task to dig around a certain section of the code for stuff that could be improved?
If you mean to talk about the tasks in a future dev dairy then you are free to wait until then and just count me an interested audience member

For research, an answer from a content designer would probably make the most sense. My understanding though is that they split tasks in two; a research task and an implementation task.

For optimization, that's generally one big task just outlining how much of an improvement to game speed we're looking for.
When we did the optimization pass last summer, I went hunting all over the code base, while the other two programmers looked at much more specific performance issues. Gwen for example looked at how to make the autosave process quicker (among other things).
The vast majority of tasks however are given to a single person. If that person runs into a wall (for example, a content designer discovering that a code change is needed), they then reassign it to someone else.

 

[blackninja9939]

This was very interesting to read, @Meneth! As someone who has just recently worked with these tools in university, it was cool to read how an actual company does it.

But I kind of wonder: one issue we had during our sprints was assigning things to tasks that aren't generally divisible into tasks. One was research (figuring out how to use Cucumber in our case). And one I could imagine to come up for you is optimisation. A lot of work on that discipline was done before the Reaper's Due patch. So I want to ask: how do you divide something vague like "optimisation" into tasks? Is it like, everybody gets a task to dig around a certain section of the code for stuff that could be improved?
If you mean to talk about the tasks in a future dev dairy then you are free to wait until then and just count me an interested audience member

Research is generally split into a separate Task, at least for all the ones I've been assigned so far that require research

 

[Meneth]

I've got a question for the developers: How many computers do you have running observation games, of which you evaluate the AI:s general behaviour? And what tools do you use moreover visually evaluating the games?

Personally, if I was working QA at Paradox I'd set up 15 computers which all run observation games on the same patch version, and constantly evaluate and prioritize the orderly evolution of a reasonably rational game world. It wouldn't be impossible to develop an algorithm with which you can use the savegames as inputs to get a multivariate analysis of each game, and get as an output a numerical value between 0 and 100 where points is added and subtracted according to a couple of factors:

- Amount of balanced independent realms and their blob factors
- Amount of balanced active religions and their respective MA:s
- Amount of particularly outlying characters regarding level of wealth, prestige & piety
- Amount of isolated counties and duchies belonging to non-merchant republics.
- Size of the emerging "Clone factor". This would reflect the proportion of living characters which belong to a cluster with the same religion, culture and traits. Using this graph as an example, a good game would have a high amount of clusters, instead of few very large and compact clusters (which would be a game where 50% of the characters are Sunni Mongols for example).
- etc.

Do you have any tools like this? I think it would be very useful approach to create the emerging story worlds CK2 is made for!

Hire me and let me develop it for you

We run overnight observation games every night. Generally around a dozen.
The main purpose is to find crashes, but we do also use the results to look for balance issues.

 

[Meneth]

@Meneth ; If a crash is only priority 1, what would constitute a priority 0 bug?

A crash that's not just a one-off, but is instead almost unavoidable.
So basically any issue that's a significant blocker for development.

 

[Meneth]

Bit of an off-topic question, but I have to take this opportunity to ask: What's the weirdest thing you've ever seen in an observer game?

Probably the 500 year old patrician (without the immortal trait; him surviving was a bug that was fixed in 2.7) with basically all the bad traits.
QA has seen weirder though.

I suspect this question might be better suited for the QA team, but how do you evaluate the results when it comes to balance? Assuming that you run observer games for hundreds of years, a fairly minor series of events early on could have huge consequences towards the end, so how do you figure out what needs to be adjusted? By inserting log outputs? Studying title history and trying to make out what wars/inheritances happened?

Edit: Also, how do you determine how ahistorical is acceptable for the average game or how large a number of "weird" outcomes is fine? Do you have any specific things you look at?

It's a mix of things. For individual features, logging or global variables are used to see that stuff makes sense.
We also look at the overall end borders. A single game going weird we won't act on, but we notice trends from time to time. If something is especially weird, we might look into how it got to be that way.

As long as there's a lot of variation and no trends are too strong, it doesn't matter that much how historical the results are; it is more important that it be interesting to play.

 

[blackninja9939]

Interesting and comprehensive DD, thanks @Meneth

Would be cool if the community could have an insight into the bug database and the possibility to vote how troublesome/annoying different bugs are,
so QA could consider this when prioritizing these different bugs.

By the way, the 'mass prisoners sell out' - function isn't working correctly for me.
Playing on CK2 'All but Sunset Invasion' 2.7.1 BPAH (only some graphical mods) and try to sell out 10 or more
people, only one of them is bought out. The other nine prisoners belong to same liege which has enough money,
but even when some ingame days/weeks passed nothing more happens.

If they belong to the same liege the game only handles one prisoner at a time per button press. So you would need to do the mass action 10 times to sell out all 10

 

[Katzura]

Interesting stuff!

I am currently involved in some software QA and have seen a similar process (Agile) at work (at work). Regarding finding optimization issues, that task is usually left to QA to log defects following general guidelines (page takes longer than 10 seconds to load, etc), and are later handed over to devs. QAs would log these defects over the course of normal testing instead of hunting them down. Then again, we work on different sorts of applications (with a lot more personnel).

Regarding this there are two different ways we handle "optimization" issues we find, we tend to keep an eye on performance and flag as soon as we see it dropping to levels that we are not happy with often resulting in optimization from devs, but if we were to have a 10 second delay on any action we would bug it straight away and at that point we would pinpoint what action it is that is taking that amount of time.

 

[blackninja9939]

Someone locked meneth in the office again.

You think he is ever allowed out?

 

[blackninja9939]

Do you let him go to a house arrest if he complains about how bad his prison cell aka coding room is, or do you send him straight to the oubliette?

I think you already know the answer...

 

[Meneth]

I hope the power of coffee will keep him Alive enough to work until he's allowed to go back to his primary cell.

Sadly I don't actually like coffee. I'm sustained by Coke Zero instead.

 

[blackninja9939]

found this interesting .

any truly horrific bug stories to share ?

If I remember correctly there was one bug a long while back in which Greek characters would evaluate whether to Castrate or Blind literally every person in their realm so if the Byzantines had a huge Empire it would kill the performance.

 

[Meneth]

Why on earth do you write an instruction like that :

if condition return false;

Though this way would be shorter :

return !condition;

Because that's far from the only thing in the function.
There's over a dozen instances of "if this thing, return false". Often "this thing" is a negation.

If it was just a single condition I totally agree "return !condition" would make more sense, but here it is just one of many early-outs.