• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

CK3 Dev Diary #92: Anatomy of a Game: From Report to Resolution

Hello, everyone, and welcome back the Dev Diary. Let's have a little talk about the Anatomy of a Game (Development)! I'll be your host for today, @jakeowaty, current QA Lead on Crusader Kings 3, and I'm joined by Elisabeth, known as @PDXOxycoon in these here parts - a wonderful programmer, a gentlewoman and a scholar, who you may have seen a few times already! (Editor's note: Can I mention the raw onion story?)

Today we are here to tell you about the process a bug goes through from being reported to being resolved and put into an update. Grab yourself the beverage of your choice, and enjoy this little expose on how we go about resolving bugs on Crusader Kings 3!


Bug report forum​

yJCQc4LtcsL3TfoVIneL1a__83p_v7WBJD-UNHq20PCuDb1aP_WuxddM8EN5wXFs91hsR0WisK7tT0l3dhIL8kgcQcZ-hnEeWJa7AKZcLZtOD5VXtAVBNtLy2KQayQjWS5VCGFgf

Whenever you launch the game, get yourself a your favorite gamer juice and steel your resolve for a few good hours of managing a medieval kingdom, there is a lot of stuff that happened behind the scenes to make it a memorable experience, free from issues. For a game as complex as Crusader Kings 3, this is even more important, and as the features get added to the game, it becomes more and more complicated, like a careful balancing act between economy, AI, warfare, interactions - all to ensure the experience you receive in the newest update is something worth waiting for!

We are but a few humble people, whose job is to find out what works and what doesn’t and report any and all issues to the team before the game hits your platform of choice. But what happens when sometimes issues do remain hidden away and they will take spitting behind your shoulder 3 times on a 3rd fortnight of a 5th month on a leap year, during a full moon while dancing hokey-pokey to uncover?

This is where you can come in and help! We can get rid of serious, big issues, find them ahead of time before you even realize they existed in the first place, and try and deliver a polished product. For anything else - well, it’s the economy of scale. With the variety of systems, languages, experiences, playstyles - YOU are the biggest help of all, in how a game can be shaped. And for that we have a special platform, called Bug Report Forum, where you can air out your grievances, provide feedback, reproduction steps and situations that irk you on the constant, so that our QA team can go through them much more efficiently and supply the team with a steady stream of new bugs to iron out for the next bug fix patch or release.


Verification​

image5.jpg

[Comic on logging]​

Once you take your valuable time to report the issue that ruins your mojo when playing Crusader Kings, it lands on our Forum as a thread. We then take turns, employing QA power to go through the forums and reproduce them locally on our machines to ensure if the issue is serious, or perhaps we can offer support by providing advice on how to avoid it.

Once the ticket is verified internally, using your reproduction steps, your description, screenshots, crash logs and saves, we then take the time to translate it into a language that our internal team can understand - Jira.

There’s an art to this. Jira is an extremely helpful and smart tool to use, and QA by nature becomes naturally adept at using it skillfully. It’s usually the QA who does project-wide database evaluations, scrubbing it from obsolete tickets and updating any and all lingering ones to make sure the project team can focus on what’s important - making more fun stuff!

Wait… Obsolete? You mean you just CLOSE old tickets?

We do… sometimes. For example, if we had an issue lingering around in some hidden away system for a long time (and sometimes not even you, our community, are able to find it for years on end) - the issue becomes obsolete - either we just make the bug into a feature, or because we change a system entirely - it becomes changed so it’s not even a problem anymore. Happens on occasion, and even we are quite surprised that some bugs that linger for long never get picked up by the players, but hey - bugs are features too! Right?

Riiiiiiiiiight…?


Jira - or how I learned to stop worrying about my mental health and learned to embrace the madness​

image2.png

[Excerpt from “How to JIRA” guide]​

Now that might look like a lot, but trust me - there’s a way to learn this. First of all, it’s pretty straight forward; complex, but not complicated. We have been campaigning for the team to use Jira tickets efficiently. For example, if we get a ticket that says “Fix the thing we talked about yesterday” and there is nothing else in it, how can we remember what we have done, what needs to be done and how do we go back to it to retest? And again - fast-forward time to a month later and believe me - nobody will remember what in the world such a ticket meant.

As a result, making sure all the fields are appropriately filled out is paramount to an efficient use of the database, and to make our team’s lives just a little bit easier when having to deal with a big pile of bugs that come from you or from our internal reporting.

If everything is sorted, and everyone is on the same page, it makes it a lot easier for the producers to schedule when and how to report. And the upvoting on the forum? That’s an incredibly powerful tool to let us know which tickets need to be prioritized and fixed, no matter what. That’s how we can influence what is important to be fixed first.

image12.png

[QA chasing the Programmers]​


The (un)lucky Programmer/Designer​

Once the bug has been verified, put into Jira, assigned a version and prioritized, a Developer can pick the bug up for fixing. This is typically either a Designer or a Programmer, depending on where the bug originates. In some cases the bug is reassigned to another role if the bug is found to be caused in another part of the game. Now that we know what the issue is and how it’s experienced, we need to investigate the how and why it happens.

As an example of this, let us look at the recently fixed issue where the Holy Orders would raise themselves and immediately disband if they were called into a war. This issue was a side effect of letting Holy Orders raise themselves for Great Holy Wars: whenever a Holy Order is at war as an ally (which they qualify as during the Great Holy Wars), they would raise their armies.

We start by looking at why the Order is disbanding the armies. If they’re raised, they must have something to fight against, right?


image7.png

[Code controlling if a order’s armies should be disbanded]​

Through developer magic known as debug mode and source code, we can step through each individual function call to see what’s going on. Stepping through this function leads us to finding that the war they are fighting in is not relevant to them, causing them to disband. Looking at the enemies in the war they’re in, none of them are of a hostile Faith. We also see that they are hired by the Grandmaster of the Order, so the Order itself decided to raise their armies, not another character hiring them.

Holy Orders will not fight in wars where the opposing side does not have any participants of hostile Faith. Because we found the Holy Order disbanding due to being raised against an irrelevant enemy, we must look at the logic which controls how they are raised.

I will spare you the initial dig down the AI code, but the short of it is: if a Holy Order can be raised, raise it. So we need to see why the Holy Order can be raised when it clearly should not be able to. We arrive at the following function.


image9.png

[Code to check the Holy Order can be hired by a Character]​

This logic checks if the Order can be hired by a character. This goes for any character looking to use the Order in question, even the Grandmaster of the Order. The Grandmaster of the Order is part of the war, so let us see where it leads us. Starting from the top, we check if the character can use the Order. Let us take a look at how that goes.


image7.png

[Code to check if a character can use the Holy Order]​

This is the code that checks if a character can use a Holy Order. When the Holy Order joins a war on its own, pCharacter is the Grandmaster, which is the Owner of the Holy Order. Makes perfect sense that the Grandmaster can make use of his own Holy Order. But, make notice further down: there is a check there, verifying if the character is in a relevant war. This is the same function that caused the army to disband earlier.

Returning back to the function which checks if a Holy Order can be hired by our Grandmaster.


image4.png

[Code to check the Holy Order can be hired by a Character 2]​

To simplify things, I have collapsed all the irrelevant parts; contained within each “if” section is something which will cause the function to fail, and not allow the Grandmaster to hire their own Order. Let’s go through these relevant parts. We don’t run afoul with the hire limit, as we don’t hire any other Holy Orders as the Grandmaster of one. We are not already hired by someone else, so we skip that part. We own this order, so we don’t touch the third “if” statement. Finally, we can afford to hire our own Order. Look at that, this means we can hire the Holy Order!

But returning back to the finding that the Holy Order would immediately disband because we’re not in a relevant war. This is where we find our solution: if we own our own Holy Order, we can always use it, but because of this we will not look if the war we are in is relevant at all. So we have a very simple solution to the problem.


image13.png

[Solution to preventing the Holy Order from being raised]​

We know that the check for relevant wars is only skipped if we are the Grandmaster of the Order, we don’t need to check the faith of our own Grandmaster, so this is what we’re left with: if the one who wants to hire the Holy Order is the Owner of the Order, check if they are in a relevant war or not.


The merge request process - everyone judges you​

When the fix has been created, it is put into what is called the merge request; this is the step before the fix is put into the upcoming build. Before it gets merged, there are a bunch of steps. It starts with one or more members of the team of appropriate discipline reviewing the changes being applied. This is the first step to spotting any unintended side effects that may occur with any given fix. The scrutiny of these reviews increase drastically with several more steps if the fix will go into an upcoming release.


image8.png

[Merge request overview for Preventing Holy Orders from raising themselves in irrelevant wars]​

Merge requests have a template that must be filled in when filed. This process gives both the fixer and reviewer the chance to review the implications of a fix and whether or not to actually commit it to a particular version. We make a serious consideration on how risky a fix is, can the fix have unintended knockon effects and the like, and how it impacts the overall performance of the game, does the fix contain a lot of computationally heavy code or script that will degrade performance.

The golden rules of managing a merge request are:
  1. Reviewer is right until proven wrong.
  2. Discipline leads are always right.

Adding a fix to a build generally requires approval from the discipline lead, tech lead and another reviewer, and only the tech lead can actually merge the fix into the build when we’re getting close to release.


image11.png

[The commit fixing the issue]​

The reviewer(s) give feedback on the merge request, and these must be addressed in one way or another. In the overview, you can see that the fix ended up being five commits in total, where just one of them was the fix itself. The rest were either addressing comments made in the review process or issues raised by the automated build and testing system, which we call pipelines. Speaking of pipelines.


image10.png

[Pipeline for merge request]​

Pipelines verify the integrity and quality of the code, build the game and test the game while the review process is going. All of these must pass for the merge request to be allowed to merge, unless explicit permission is given from the programmers, and in the case of an upcoming release the pipelines must be green. No exceptions.

Once the fix has been merged, we hand it back over to the sadistic wonderful QA team.

The (sadistic) QA verification - where QA asks the fixer: “why are you running?!”​

image6.png

[Happy tears from QA after a 2 year old issue was resolved]​

Once a ticket gets processed, churned through the complicated development machine, the issue is removed from the game, the correct code, script or asset gets merged into the game, the ticket is then marked as resolved.

This is when we get our hands on it one more time to verify if the fix has not only resolved the issue listed therein, but also that it has no other knock-on effects. Think of big fixes like a domino - you push one piece and the others will fall. Sometimes a different set of dominos, that we set up a long time ago, gets randomly pushed and some older issues reappear out of nowhere. It happens when you deal with a complex codebase, in which case you sometimes do see returning fan favorites.

But all we have to do is revisit the older system, fix it (again), and away we go! Unless yet another bug is retriggered, in which case… Well, we have to revisit yet another system. Videogames are hard, m’kay?

This is why we sometimes need to go through the same ticket again, and again, and again, until we are very sure it fixes more things than it breaks. Sometimes bugs linger in our database for a long time. Trust us - we know that issues you mentioned on release still persist and we want to fix everything, but with every release we get a new batch of tickets we have to go through…

So.
Many.
Times.
However, once the fix is confirmed, and the build is locked in - we are ready to press the large green button and release it for your scrutiny and repeat aforementioned hokey-pokey jig again!


The Update - Are we done yet?​

Not really. On our end we close the bug as resolved, but there is one last thing that comes into play on resolving a bug. This is all of you, the community. The QA team catches most of the problems before a release is made, but sometimes for an issue to reoccur, you need the stars to align on a laptop manufactured on 29th of February 2020 while the player is doing the Macarena.

Paraphrasing, of course, but there are cases where an issue can reemerge later. This can happen due to a myriad of factors that are hard to account for. Some issues can be related to hardware and specific circumstances we don’t have the capacity to test for. This is when we rely on you all to be vigilant and report issues back to us. The more detailed and specified, the better. Once this is done we go back from the top of this expose and do the dance one more time.

If no one in the community experiences the issue again, then it stays resolved.

I hope you all have enjoyed this little expose into the life of a game developer! Thank you for your time, and we wish you a very pleasant day.

Until next time!
 
  • 87
  • 42Like
  • 18Haha
  • 12
  • 9Love
  • 2
Reactions:
To be honest, I prefer reporting bugs directly to Discord with a quick text and screenshot of the bug rather than filling the form on the forum. I've found too many bugs and yet I am too lazy to open threads for each one of them. I would gladly report it through discord tho.
 
  • 55
  • 9
  • 5Haha
  • 3Like
  • 1
Reactions:
The bug report thread would be much more streamlined if the search function wasn't as messy to use. But this is a pretty cool post overall.
 
Last edited:
  • 6Like
  • 3
Reactions:
Great stuff ! Love to sneak peaks backstage.

Question about the bug report forum: there are long periods of time (sometimes months) when bug reports aren't flagged, which always leaves users wondering whether they are being parsed at all, and if it's even worth reporting more bugs. Any reason for this, and should we keep reporting?

Bonus question: what's the proper etiquette when QA flags a bug report As Designed, and you're pretty sure it's not so? A report I have in mind was about a Varangian Adventure war after which the character that used the CB kept one of its initial holdings - which from what everything the game says about that CB, shouldn't happen.

And yet another one: there are instances where reports are flagged As Designed, but where there's likely a flaw in the design itself. If it's flagged As Designed, it'll likely never be looked at further. What's the proper way to report those? On one such report, it was suggested I post it in Suggestions, but we have no idea what's your process in handling those, and there's no feedback at all (which doesn't really encourage using them).
 
  • 15
  • 6
  • 3Like
  • 1
Reactions:
Interesting thread. Thanks for explaining the process of the bug report.
 
  • 2
Reactions:
Hello, could you fix the banner editor in russian localization? Problem is that when editing any figures, the sliders go off the screen.

I really love the game, soon I will get all the achievements!
 
To be honest, I prefer reporting bugs directly to Discord with a quick text and screenshot of the bug rather than filling the form on the forum. I've found too many bugs and yet I am too lazy to open threads for each one of them. I would gladly report it through discord tho.
I disagree, simply because Discord is a realtime medium of communication and ill suited for going back to see and respond to archived topics. Due to the short message nature of Discord, it's also hard to get proper details about something if there are also other ongoing discussions within the same channel at the same time. The forum is much better suited for this.

Question about the bug report forum: there are long periods of time (sometimes months) when bug reports aren't flagged, which always leaves users wondering whether they are being parsed at all, and if it's even worth reporting more bugs. Any reason for this, and should we keep reporting?
I cannot really say much in regards to the bug reporting forum, as I grab my tickets from *shudder* Jira, but absolutely keep reporting. If you cannot find a thread with what you're experiencing, make a thread for it! We have an internal document that is regularly updated with top issues from the community, and these do affect which tasks I select from Jira when I go searching for tasks to do.

Bonus question: what's the proper etiquette when QA flags a bug report As Designed, and you're pretty sure it's not so? A report I have in mind was about a Varangian Adventure war after which the character that used the CB kept one of its initial holdings - which from what everything the game says about that CB, shouldn't happen.
Personally I think raising the question again within the thread, and reiterate upon the steps required to produce the outcome that you believe is faulty and what you believe should be the desired outcome. Some times this all falls down to a misunderstanding or miscommunication somewhere in the chain.

Are you using capital letters for both classes and methods? .NET background? :)
I don't know the background of it for PDS, but I'm quite an enthusiast when it comes to clear and consise standards for writing code within an organization ;)
 
  • 25
  • 9Like
  • 1
Reactions:
Very insightful DD. It can definitely be frustratring when bugs we report seem to be ignored, but this reveals that sometimes the process is a bit more complicated than it intially seems.
 
  • 6Like
  • 1
Reactions:
Personally I think raising the question again within the thread, and reiterate upon the steps required to produce the outcome that you believe is faulty and what you believe should be the desired outcome. Some times this all falls down to a misunderstanding or miscommunication somewhere in the chain.

Slight problem with this is that threads marked 'As Designed' are usually locked immediately?

And yet another one: there are instances where reports are flagged As Designed, but where there's likely a flaw in the design itself. If it's flagged As Designed, it'll likely never be looked at further. What's the proper way to report those? On one such report, it was suggested I post it in Suggestions, but we have no idea what's your process in handling those, and there's no feedback at all (which doesn't really encourage using them).

Which in these cases can just be a bit frustrating? I'll use an example of my own:


With the Royal Court patch, the ability to rightclick counties on the character select screen to view their lord's education traits and skill points at a glance was outright removed, but you can still see it by using the character finder, which just feels like an awkward workaround to a problem that never existed. In this case, it appears that characters were never meant to be able to be rightclicked in the character select screen, although I can see no benefit to them not being able to and having this feature removed, but I couldn't argue my case because the thread was closed immediately. Would appreciate some insight on this.

Just adding an addendum that this dev diary was pretty much exactly what I've been looking for in terms of "Dev Diaries When Content Updates Are Otherwise Sparse", and while it's not as layman-friendly as the Outer Worlds devs explaining how they fixed a pervasive bug (https://www.polygon.com/2019/12/13/...s-companion-bug-dead-qa-taylor-swope-obsidian) (really great read by the way for game devs and non-devs alike), I'm happy with the direction transparency is going in.
 
  • 2Like
Reactions:
Great stuff ! Love to sneak peaks backstage.

Question about the bug report forum: there are long periods of time (sometimes months) when bug reports aren't flagged, which always leaves users wondering whether they are being parsed at all, and if it's even worth reporting more bugs. Any reason for this, and should we keep reporting?

Bonus question: what's the proper etiquette when QA flags a bug report As Designed, and you're pretty sure it's not so? A report I have in mind was about a Varangian Adventure war after which the character that used the CB kept one of its initial holdings - which from what everything the game says about that CB, shouldn't happen.

And yet another one: there are instances where reports are flagged As Designed, but where there's likely a flaw in the design itself. If it's flagged As Designed, it'll likely never be looked at further. What's the proper way to report those? On one such report, it was suggested I post it in Suggestions, but we have no idea what's your process in handling those, and there's no feedback at all (which doesn't really encourage using them).

So while we'd love to have a permanent presence on the forum, currently we ensure that we do a lot of coverage around patches - that's when we usually have a huge influx of players eager to test out new content, which coincides with higher report rate on the forum.

In the past we used to be able to do it more regularly, but because our process is becoming more and more complex, our workflows complicated and QA is now involved in almost all parts of development like a helix, we have less time to do it ourselves.

But that doesn't mean we don't do it - we simply concentrate efforts to go through current and some of the backlog.


If you are very particular about certain bugs and you'd like them desperately fixed, you can help us locate and see them by upvoting threads with bugs you are very interested in. Usually each release in subsequent weeks we pick out top upvoted issues from the past week or two, select a few pages of them and make sure all of them are fixed.

Hello, could you fix the banner editor in russian localization? Problem is that when editing any figures, the sliders go off the screen.

I really love the game, soon I will get all the achievements!

I'm pretty sure we fixed this one. Heavens, is it broken again?
 
  • 14
Reactions:
Slight problem with this is that threads marked 'As Designed' are usually locked immediately?

That shouldn't be the case, but if it is - feel free to report these to the respective QA who have locked the thread to keep them unlocked. We try to keep the conversation flowing, and sometimes it's not the QA is speaking, but we simply are the extensions of our internal dev team answering questions for you.

Please don't be getting mad at us! (Also, I unlocked your thread ;))
 
  • 19
  • 3Like
Reactions:
If you are very particular about certain bugs and you'd like them desperately fixed, you can help us locate and see them by upvoting threads with bugs you are very interested in. Usually each release in subsequent weeks we pick out top upvoted issues from the past week or two, select a few pages of them and make sure all of them are fixed.

I certainly didn't realize that upvoting bug reports was a worthwhile activity, and, based on the fact that most bug reports get 0-2 votes, I suspect most other users didn't either. It feels like the main determiner of the number of votes a bug report gets is whether or not it is linked to in a thread in the main forum, not the severity or frequency with which players encounter it.
 
  • 13
  • 1Like
Reactions:
I certainly didn't realize that upvoting bug reports was a worthwhile activity, and, based on the fact that most bug reports get 0-2 votes, I suspect most other users didn't either. It feels like the main determiner of the number of votes a bug report gets is whether or not it is linked to in a thread in the main forum, not the severity or frequency with which players encounter it.

Yeah I usually just determine whether a bug has been noted (or the likelihood of this) by whether or not it has a dev response.
 
  • 6
  • 1Like
Reactions: