Good afternoon, all. I’m Magne “Meneth” Skjæran, the programmer on CK2. In the past I’ve written dev diaries about modding, optimization, and quality of life improvements, as well as last week’s dev diary talking about what each role on the team is responsible for.
Summer is still ongoing, and most of the office is gone until the end of July, including the majority of the CK2 team. The show must still go on, so for these four weeks I’ll be writing most or all the CK2 dev diaries.
Since most of the team being on vacation means that little development is happening, that means that these four dev diaries will be pretty light on info about the upcoming expansion and patch, but I hope to provide you with some interesting insights, info, and tidbits regardless.
Today, I’ll be talking about the life of a bug; essentially a view into how a bug goes from discovery to release of a fix, and who is involved at each stage.
At the end of this diary, I’ll also share five changelog entries from the 2.8 patch chosen (mostly) at random.
The Bug
To illustrate the process, I’ll be using the following bug as a case study of sorts:
- Fixed it in some cases being possible for priests of religions where priests are not allowed to marry to marry or betroth someone
This bug fix was included in the 2.7.1 patch. What it essentially did, was fix the Pope ending up with a family member that loves Divine Blood marriages so much, they’d betroth the Pope, even though he’s not allowed to marry. Xwedodah is a powerful thing, but this would always end with tears: the moment the Pope’s niece came of age, the betrothal would get broken.
In retrospect, this changelog entry should clearly have been “Fixed the Pope loving Xwedodah so much that he’d get betrothed despite not being allowed to marry”. You live and learn.
Discovery
This bug was discovered by one of our beta testers on 12th February, 2017 while playing the game; they noticed that the Pope would get matrilineally betrothed to his niece, and had a save from before this where it would consistently happen.
Discovery by beta testers is however only one of many avenues of discovery. Other ways include:
Playtesting: Quality Assurance testers spend some of their time simply playing the game, though usually with some kind of focus based on what we’re working on. For example, during Monks and Mystics development the focus might have been participating in a specific society. During playtesting, they’ll report any bugs they run into, as well as making note of balance issues, usability issues, and similar. Depending on the stage of development, these will also be reported directly; if the system is a work in progress, the developer responsible might simply be poked with the issue, but if it is supposed to be finished, a bug report is generally made.
Task verification: When a task (for example, “add a prisoner mass action system”) has been completed, quality assurance will verify that it works according to specification. If they find issues during this time they will either reopen the task, or make bug reports about the task. You can read more about verification in the “Verification” section of this DD.
Non-QA developers: The non-QA devs play the game as well, and are expected to report any bugs they encounter, even though finding bugs is not their primary purpose on the project.
Forum monitoring: While the other bug discovery methods are focused on discovering issues before the patch is released at all, things do slip through the cracks. When they do, they’re often discovered by regular players of the game, some of which end up making bug reports on the forums, on reddit, or similar. Quality assurance spends some of their time monitoring these places, making internal bug reports when issues can be replicated. If the bug cannot be replicated based on the information provided, they will usually ask for more information. An internal bug report will only be made if either the issue can be replicated, or a lot of people are having it. So if you want to make sure your bug report has the best possible chance to result in a fixed bug, you should always provide reproduction steps and a save if at all possible!
Bugs are found in a number of other ways as well, but these four avenues and beta testing cover the majority of cases.
Reporting
Once a bug has been found, it has to be reported internally. Our bug database is contained in the project management web application Jira, which provides a variety of ways to easily manage issues so that they can be scheduled appropriately and things don’t slip through the cracks.
Above you can see the report for the Pope betrothal issue.
The main things needed when reporting are:
Title: A title summarizing the issue. This is supposed to follow the format: Project - Expansion - Area affected - Description of the issue. In this case, “CK2 - MnM - AI - Pope gets engaged matrilineally to the daughter of his kin king”. The title should make it clear at a glance what the issue is and when it was discovered.
Description: The description includes any details that can’t be fit into the title. This often includes a description of what the player was doing when running into the bug, theories about the cause, or in the case of bugs discovered via forum monitoring, the source of the bug report.
Steps to reproduce: In some ways the most important section, the steps to reproduce says how the issue can be replicated. Without this, most bugs are difficult to impossible to fix, as just the information that something can happen often isn’t enough to figure out how it can happen. This is used by both the person fixing the issue (see “Fixing”) and the person verifying the fix (see “Verification”).
Result: Reports also include both the expected result and actual result, to make it even clearer what the actual issue is. Often it is relatively obvious from the other fields, but especially for things like balance issues, these sections can be of great help. They also make it easier to figure out if a problem has actually been fixed or not (see “Verification”).
Attachments: Often a screenshot of the issue is attached. A save is usually included as well if that makes reproduction easier. For crashes and similar, logs are also included, as well as a crash dump which can be used by the programmer to see where in the code the game crashed.
There’s a number of fields beyond this, but they’re generally set later in the process.
Assignment
Once a bug has been reported, it has to be decided who is responsible for fixing it, how important it actually is to fix, and when it should be fixed. Responsibility is first done by discipline; a bug will be assigned to Script, Code, or Art, depending on what causes it. This assignment is usually done by the quality assurance testers if it was not set when the bug report was created.
Priority is set depending on how important a bug is to fix, and ranges from 0 (Must Fix Immediately) to 5 (No Priority). A crash for example would be priority 1 - Must Fix, while the Pope betrothal bug was simply priority 3 - Medium. The project lead has the final call on priority, but generally quality assurance will set the priority for bugs, while the project lead prioritises tasks.
Similarly, there’s Severity, which ranges from Blocker to None. Rather than being how important fixing the issue is, this represents how broken it is. The two correlate to some extent, but you could have something like a highly visible spelling error being High Priority while being only Minor Severity. Severity is set by QA rather than the project lead.
Then, it has to be decided when the bug should actually be fixed. This is done by assignment to a sprint (a period of time during which a specific set of tasks are to be completed. On CK2, these last 4 weeks). For important bugs, they’re generally assigned to the sprint currently ongoing, while less important ones are generally left for the polish weeks rather than assigned to a regular sprint. The exception is bugs relating to new features; those are nearly always assigned to the ongoing sprint even if they aren’t especially severe. This is because it is considered more important to ensure new bugs aren’t added than to fix bugs that are already in the released product. However, the older bugs still do get assigned to the polish weeks, once the main work on the expansion itself is done. During feature sprints, the last of four weeks is dedicated to fixing bugs; during the first three weeks generally only bugs that block other work will get fixed.
When the sprint with the bug comes along, the bug is then assigned to a specific person. The project lead has responsibility for this, but it is often delegated to the discipline itself. For example, the content designers generally divide bugs between themselves autonomously rather than having the project lead assign them.
Fixing
All issues in the sprint more important than the bug have been resolved, so the time has finally come to fix it. The person assigned the bug will now hit “Start Progress” on the bug, thus informing the rest of the team of what they’re working on.
The person will then go through the steps to reproduce, and try to figure out where things are going wrong. In the case of the Pope betrothal, the bug was assigned to me, and the steps to reproduce worked flawlessly. This allowed me to set a “breakpoint” in the code for the AI marriage logic, pausing the code execution when someone would betroth the Pope. This allowed me to work upwards until I found where in the code the theocratic requirement on marriage was located. In the end, it turned out the “CanMarry” function never actually checked for this, only the marriage interaction itself did. As such, betrothals could still happen, but would get invalidated when the characters involved came of age and tried to marry.
End result was one new line of code in the CCharacter::CanMarry function:
This simply checks if either character is a theocrat belonging to a religion that doesn’t allow priests to marry.
After recompiling the game, I then ran the steps to reproduce from the bug report once more, and saw that the betrothal no longer happened.
Once the fix itself has been made, a changelog entry is written. The fix and changelog entry are then committed to our repository for the game. Our build servers then compile the game, and a few hours after midnight, a new version of the game is uploaded to Steam for our beta testers to play.
The bug is then marked as resolved as “Fixed” with a message summarizing what was done, and at what revision the change was made. In the case of the Pope betrothal, it was marked as fixed 6 days after it was reported, but this can easily vary from minutes to months.
Verification
Next up, quality assurance verifies that the bug is actually fixed, and that new issues in the area haven’t appeared as a result.
This is done by following the reproduction steps to see if the issue still occurs, and if the issue is especially complicated, trying to reproduce it in other ways.
As mentioned in the “Discovery” section, quality assurance also does playtesting to discover new bugs, and this often includes bugs that are unintended side-effects of fixes to other bugs.
When QA is satisfied that the bug is indeed gone, they mark the issue as closed, and the fix process is typically over. If the bug actually still persists in some form, the bug is reopened, and we return to the “Fixing” step.
Sometimes a bug will resurface even after verification. This generally leads to the issue being reopened and fixed once more.
In the case of the Pope betrothal, the bug was verified 45 days after it was resolved as fixed. We try to avoid having verification lag more than a sprint or so (a month) behind resolution, as it is generally easier to go back and fix something properly if you worked on it recently.
Release
The bug has been fixed, and it is time for a patch, but the process isn’t quite complete yet.
While the specific bug has been verified as fixed, the project as a whole needs to go through one last step to be considered ready to release: the smoke test.
The smoke test essentially checks that nothing is horrifically broken on the three operating systems we support: Windows, Mac, and Linux. Unlike the other steps in the process, the smoke test is not conducted by the CK2 team, but instead by PDS’ central QA team.
Once central QA has signed off on the release, no more changes can be done. Some days to weeks later, the patch is released, and the bug is now fixed for all players.
Summary
All in all we have a number of stages to the bug fixing process to ensure that bugs are found, fixed, and stay fixed. Despite this, bugs do slip through the cracks as CK2 is a rather complicated beast. There’s also always known bugs in each release, as at some point release has to happen, and fixing one bug can easily cause another. We try to minimize the number of known bugs by having a long period of time dedicated to polishing the game before each release, and by having a shorter period during which no changes are allowed at all except fixes to critical issues. There will always be bugs, but we’re always working hard to avoid introducing them, and reduce the number of them.
I hope you found this look into how we handle bugs at CK2 informative.
Now as promised, here’s five changelog entries from the upcoming (no public ETA at this time) 2.8 patch:
Summer is still ongoing, and most of the office is gone until the end of July, including the majority of the CK2 team. The show must still go on, so for these four weeks I’ll be writing most or all the CK2 dev diaries.
Since most of the team being on vacation means that little development is happening, that means that these four dev diaries will be pretty light on info about the upcoming expansion and patch, but I hope to provide you with some interesting insights, info, and tidbits regardless.
Today, I’ll be talking about the life of a bug; essentially a view into how a bug goes from discovery to release of a fix, and who is involved at each stage.
At the end of this diary, I’ll also share five changelog entries from the 2.8 patch chosen (mostly) at random.
The Bug
To illustrate the process, I’ll be using the following bug as a case study of sorts:
- Fixed it in some cases being possible for priests of religions where priests are not allowed to marry to marry or betroth someone
This bug fix was included in the 2.7.1 patch. What it essentially did, was fix the Pope ending up with a family member that loves Divine Blood marriages so much, they’d betroth the Pope, even though he’s not allowed to marry. Xwedodah is a powerful thing, but this would always end with tears: the moment the Pope’s niece came of age, the betrothal would get broken.
In retrospect, this changelog entry should clearly have been “Fixed the Pope loving Xwedodah so much that he’d get betrothed despite not being allowed to marry”. You live and learn.
Discovery
This bug was discovered by one of our beta testers on 12th February, 2017 while playing the game; they noticed that the Pope would get matrilineally betrothed to his niece, and had a save from before this where it would consistently happen.
Discovery by beta testers is however only one of many avenues of discovery. Other ways include:
Playtesting: Quality Assurance testers spend some of their time simply playing the game, though usually with some kind of focus based on what we’re working on. For example, during Monks and Mystics development the focus might have been participating in a specific society. During playtesting, they’ll report any bugs they run into, as well as making note of balance issues, usability issues, and similar. Depending on the stage of development, these will also be reported directly; if the system is a work in progress, the developer responsible might simply be poked with the issue, but if it is supposed to be finished, a bug report is generally made.
Task verification: When a task (for example, “add a prisoner mass action system”) has been completed, quality assurance will verify that it works according to specification. If they find issues during this time they will either reopen the task, or make bug reports about the task. You can read more about verification in the “Verification” section of this DD.
Non-QA developers: The non-QA devs play the game as well, and are expected to report any bugs they encounter, even though finding bugs is not their primary purpose on the project.
Forum monitoring: While the other bug discovery methods are focused on discovering issues before the patch is released at all, things do slip through the cracks. When they do, they’re often discovered by regular players of the game, some of which end up making bug reports on the forums, on reddit, or similar. Quality assurance spends some of their time monitoring these places, making internal bug reports when issues can be replicated. If the bug cannot be replicated based on the information provided, they will usually ask for more information. An internal bug report will only be made if either the issue can be replicated, or a lot of people are having it. So if you want to make sure your bug report has the best possible chance to result in a fixed bug, you should always provide reproduction steps and a save if at all possible!
Bugs are found in a number of other ways as well, but these four avenues and beta testing cover the majority of cases.
Reporting
Once a bug has been found, it has to be reported internally. Our bug database is contained in the project management web application Jira, which provides a variety of ways to easily manage issues so that they can be scheduled appropriately and things don’t slip through the cracks.
Above you can see the report for the Pope betrothal issue.
The main things needed when reporting are:
Title: A title summarizing the issue. This is supposed to follow the format: Project - Expansion - Area affected - Description of the issue. In this case, “CK2 - MnM - AI - Pope gets engaged matrilineally to the daughter of his kin king”. The title should make it clear at a glance what the issue is and when it was discovered.
Description: The description includes any details that can’t be fit into the title. This often includes a description of what the player was doing when running into the bug, theories about the cause, or in the case of bugs discovered via forum monitoring, the source of the bug report.
Steps to reproduce: In some ways the most important section, the steps to reproduce says how the issue can be replicated. Without this, most bugs are difficult to impossible to fix, as just the information that something can happen often isn’t enough to figure out how it can happen. This is used by both the person fixing the issue (see “Fixing”) and the person verifying the fix (see “Verification”).
Result: Reports also include both the expected result and actual result, to make it even clearer what the actual issue is. Often it is relatively obvious from the other fields, but especially for things like balance issues, these sections can be of great help. They also make it easier to figure out if a problem has actually been fixed or not (see “Verification”).
Attachments: Often a screenshot of the issue is attached. A save is usually included as well if that makes reproduction easier. For crashes and similar, logs are also included, as well as a crash dump which can be used by the programmer to see where in the code the game crashed.
There’s a number of fields beyond this, but they’re generally set later in the process.
Assignment
Once a bug has been reported, it has to be decided who is responsible for fixing it, how important it actually is to fix, and when it should be fixed. Responsibility is first done by discipline; a bug will be assigned to Script, Code, or Art, depending on what causes it. This assignment is usually done by the quality assurance testers if it was not set when the bug report was created.
Priority is set depending on how important a bug is to fix, and ranges from 0 (Must Fix Immediately) to 5 (No Priority). A crash for example would be priority 1 - Must Fix, while the Pope betrothal bug was simply priority 3 - Medium. The project lead has the final call on priority, but generally quality assurance will set the priority for bugs, while the project lead prioritises tasks.
Similarly, there’s Severity, which ranges from Blocker to None. Rather than being how important fixing the issue is, this represents how broken it is. The two correlate to some extent, but you could have something like a highly visible spelling error being High Priority while being only Minor Severity. Severity is set by QA rather than the project lead.
Then, it has to be decided when the bug should actually be fixed. This is done by assignment to a sprint (a period of time during which a specific set of tasks are to be completed. On CK2, these last 4 weeks). For important bugs, they’re generally assigned to the sprint currently ongoing, while less important ones are generally left for the polish weeks rather than assigned to a regular sprint. The exception is bugs relating to new features; those are nearly always assigned to the ongoing sprint even if they aren’t especially severe. This is because it is considered more important to ensure new bugs aren’t added than to fix bugs that are already in the released product. However, the older bugs still do get assigned to the polish weeks, once the main work on the expansion itself is done. During feature sprints, the last of four weeks is dedicated to fixing bugs; during the first three weeks generally only bugs that block other work will get fixed.
When the sprint with the bug comes along, the bug is then assigned to a specific person. The project lead has responsibility for this, but it is often delegated to the discipline itself. For example, the content designers generally divide bugs between themselves autonomously rather than having the project lead assign them.
Fixing
All issues in the sprint more important than the bug have been resolved, so the time has finally come to fix it. The person assigned the bug will now hit “Start Progress” on the bug, thus informing the rest of the team of what they’re working on.
The person will then go through the steps to reproduce, and try to figure out where things are going wrong. In the case of the Pope betrothal, the bug was assigned to me, and the steps to reproduce worked flawlessly. This allowed me to set a “breakpoint” in the code for the AI marriage logic, pausing the code execution when someone would betroth the Pope. This allowed me to work upwards until I found where in the code the theocratic requirement on marriage was located. In the end, it turned out the “CanMarry” function never actually checked for this, only the marriage interaction itself did. As such, betrothals could still happen, but would get invalidated when the characters involved came of age and tried to marry.
End result was one new line of code in the CCharacter::CanMarry function:
Code:
if ( ( GetGovernment()->IsTheocracy() && !GetReligion()->CanPriestsMarry() ) || ( pCharacter->GetGovernment()->IsTheocracy() && !pCharacter->GetReligion()->CanPriestsMarry() ) )
return false;This simply checks if either character is a theocrat belonging to a religion that doesn’t allow priests to marry.
After recompiling the game, I then ran the steps to reproduce from the bug report once more, and saw that the betrothal no longer happened.
Once the fix itself has been made, a changelog entry is written. The fix and changelog entry are then committed to our repository for the game. Our build servers then compile the game, and a few hours after midnight, a new version of the game is uploaded to Steam for our beta testers to play.
The bug is then marked as resolved as “Fixed” with a message summarizing what was done, and at what revision the change was made. In the case of the Pope betrothal, it was marked as fixed 6 days after it was reported, but this can easily vary from minutes to months.
Verification
Next up, quality assurance verifies that the bug is actually fixed, and that new issues in the area haven’t appeared as a result.
This is done by following the reproduction steps to see if the issue still occurs, and if the issue is especially complicated, trying to reproduce it in other ways.
As mentioned in the “Discovery” section, quality assurance also does playtesting to discover new bugs, and this often includes bugs that are unintended side-effects of fixes to other bugs.
When QA is satisfied that the bug is indeed gone, they mark the issue as closed, and the fix process is typically over. If the bug actually still persists in some form, the bug is reopened, and we return to the “Fixing” step.
Sometimes a bug will resurface even after verification. This generally leads to the issue being reopened and fixed once more.
In the case of the Pope betrothal, the bug was verified 45 days after it was resolved as fixed. We try to avoid having verification lag more than a sprint or so (a month) behind resolution, as it is generally easier to go back and fix something properly if you worked on it recently.
Release
The bug has been fixed, and it is time for a patch, but the process isn’t quite complete yet.
While the specific bug has been verified as fixed, the project as a whole needs to go through one last step to be considered ready to release: the smoke test.
The smoke test essentially checks that nothing is horrifically broken on the three operating systems we support: Windows, Mac, and Linux. Unlike the other steps in the process, the smoke test is not conducted by the CK2 team, but instead by PDS’ central QA team.
Once central QA has signed off on the release, no more changes can be done. Some days to weeks later, the patch is released, and the bug is now fixed for all players.
Summary
All in all we have a number of stages to the bug fixing process to ensure that bugs are found, fixed, and stay fixed. Despite this, bugs do slip through the cracks as CK2 is a rather complicated beast. There’s also always known bugs in each release, as at some point release has to happen, and fixing one bug can easily cause another. We try to minimize the number of known bugs by having a long period of time dedicated to polishing the game before each release, and by having a shorter period during which no changes are allowed at all except fixes to critical issues. There will always be bugs, but we’re always working hard to avoid introducing them, and reduce the number of them.
I hope you found this look into how we handle bugs at CK2 informative.
Now as promised, here’s five changelog entries from the upcoming (no public ETA at this time) 2.8 patch:
- The AI has learned how to spell, and no longer look for "soceities" missions when they're actually called "societies" missions. With this newfound ability to spell, the AI is better at identifying mission targets
- Fixed using council consideration letting you bypass the need for a favor on your liege
- Added Holy Order for Manicheans
- Guardians can now be assigned from birth (though there are no effects until age 6.)
- AI should no longer pine for a matrilineal marriage if they have been turned off by game rules
.png){kind=avatar}
