• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.
Greetings, fellow crusaders!

In this Dev Diary we’ll give a brief insight in how the CK2 QA team works and functions.

The team consists of myself (Servancour), PeterSkager, rageair and Leo Larsson. I’ve been here the longest and have been at Paradox for close to two years now, and so far it’s been a blast! We’re a close-knit team and work very well together, each and every one specializing in different types of gameplay.

As a team, we work very closely with the developers and attend all sorts of meetings to ensure that we stay up to date with the game. When you work with QA you need to know almost everything about the game, from overarching design to the smaller details of specific features. It’s challenging but also very rewarding.

While our primary responsibility may be to test the game, it’s not the only one. Many believe that the life of a QA only consists of reporting bugs, but it’s actually much more than that. We’re responsible for ensuring the overall quality of the product, by providing constant feedback, voice concerns where needed, comment on design and suggest improvements in order to make the game both better and more accessible!

Something we do every now and then when there’s new features to test is to simply play the game in order to get a feel for how to play with them. We usually test a variety of different scenarios, with varying start dates and setups, where we play for anything between a few years to several centuries. Sometimes we play for just a few hours, other times we have the campaigns going over the course of a few days, all while we gather feedback and impressions to the developers. This type of testing allows us to get a more accurate representation on how the game actually plays to find issues you wouldn’t find in a few minutes of playtime. Sometimes we even start using the same scenario just to see how far we can deviate from each other as we use very different playstyles. We believe that having many eyes on the same problem yield the best results.

We’re extremely passionate about the game. Each and every one of us here at the CK2 QA team has played the game for anything between a few hundred to several thousand hours and most of it has been purely for our own enjoyment. I myself play the game regularly outside of work and in my latest campaign I started off as Charlemagne himself, culture switched to German and created the Holy Roman Empire! I’m almost done with it though, with just the last hundred years or so left before the end date. My current ruler is Kaiserin Amalia “The Hammer”, ruling over Europe with an iron fist!

This passion also motivates us to not only ensure that the game performs well from a professional standpoint, but also from the perspective of a player. This causes us to often go above and beyond our ordinary tasks! We at QA has, on occasion, contributed with content to the game. Often to ensure that each and every part of the game feels like a quality experience. For example; we’ve created events to enhance certain aspects of the game (see the Sky Burial and Master Wrestling event chains) and even created art in the form of the Nomadic CoA’s introduced in Horse Lords.

I hope you gained some useful insight in how we approach CK2 from the QA team, and that we work hard to give you, the fans, a product that you deserve. Since we at QA love quality of life improvements, I’ll wrap this up by showing you the following screenshot of a previous feature we are extending somewhat:

ck2_diary_screen.jpg
 
This seems like a good place for

Flanders and Aquitaine for France!
Can people just shut up about the Flanders issue,... I agree with your point, but I am getting sick of seeing it posted everywhere. It is about as annoying as the bloody china bandwagon!
 
  • 13
  • 12
Reactions:
Luckily people here understand things like math. If 10,000 people play for an undefined "few" hours, lets say 3, that is 30,000 hours of testing. For 4 QA working 8 hour days that would be ~2.5 years of testing. Which doesn't even consider that QA aren't testing the final version for the entire period, or that fixing one bug can reveal other bugs, unbalanced things, or exploits.

Why would someone continue to maintain and pay for 4 QA, while releasing closed beta for around 100-200 people would be better choice(imo) ?
 
  • 6
Reactions:
Who says I haven't? :p
Seriously though, I could have conquered the world two times over most likely, but instead I pace myself when I play longer campaigns. I set these small goals for myself to do every now and then. Just to break up the regular gameplay of expanding and do something different.

Since this is the case, would it not make sense to expand the ambitions system? Or create some new functionality to create on-the-fly objectives for the player? I have bumped into the same issue, myself, where expanding becomes a grind but is often the only thing I feel is available to do.
 
  • 1
Reactions:
At the moment, we don't do a lot of automated testing. But this is definitely something we are looking to do much more in the future in order help us out with some of the more basic testing procedures.
Good luck with that. Covering a large existing code base with automated testing often feels thankless, but it is worth it when you start regularly catching new bugs as they are introduced. :)

I'm currently reading "Working Effectively with Legacy Code by Michael Feathers. I wish I read it before rather than after starting my current job. It's pretty good so far.
 
Why would someone continue to maintain and pay for 4 QA, while releasing closed beta for around 100-200 people would be better choice(imo) ?

Quality of reported bugs is the main reason. Beta testers are great, but they aren't paid to test and most of the time they'll test stuff that is fun. There are exceptions of course and some beta testers amaze us with the work they put in.
Paradox relied almost solely on beta testers up until V2 release (and then only as limited time contractors), EU4 was the first releases (as far as I remember with a in house team of QA). I'd say that quality has gone up since EU3 and HOI3 releases. QA isn't the only reason, but it's definitely one of the major.
 
  • 11
  • 7
Reactions:
Quality of reported bugs is the main reason. Beta testers are great, but they aren't paid to test and most of the time they'll test stuff that is fun. There are exceptions of course and some beta testers amaze us with the work they put in.
Paradox relied almost solely on beta testers up until V2 release (and then only as limited time contractors), EU4 was the first releases (as far as I remember with a in house team of QA). I'd say that quality has gone up since EU3 and HOI3 releases. QA isn't the only reason, but it's definitely one of the major.
The QA mindset is very different than both developers and typical players. I'm honestly amazed that Paradox put out so many games without having in house QA.

It does explain a bit about some of the issues I have with earlier Paradox games. EU4 was the most polished release I've experienced with Paradox games. That bodes well for future releases. :)
 
  • 4
Reactions:
Not really.
QA can only inform the dev-team and producers about problems, they have no mandate to cancel or post-pone a game.
If someone is to blame it's either me or the producer (or anyone above us in the chain of command).
Well, I'd say it's kinda of compicated.
First of all, it's not that I despise testers or something like that. I'm in this business exactly for nearly 15 years, so I believe I know the drill.
But to be honest I believe we're speaking different things here.
There is bugs such as "hey, this new mechanics unlock some spicy way to win, a-ha!". It's not QA responsibility - as was said already, just simple math says us users always find way better. If they catch something like that, well, they supposed to describe it to programmers with something about "suggestion" tag. No arguing here.
Also there is bugs such as "hey, it's obvious we need this feature here for hundred of years". It's not QA responsibility again, it's for dev.
But there is bugs, such as, you know, bugs. Sometimes button to reform non-norse pagan (if I recall a case right) just disappears from interface. Or maybe levies just don't dissolve. I offer some obvious bugs, with no sane developer or producer will release build with such things, that's not something you just put "do-not-fix" and ignore. Once again, that's obvious ones, but there happens something more obscure, such as "AI never build anything" or "AI builds, but too rarely". I'm trying to speak about that ones I saw in forums here, not something I noticed and ignored.
So if that bugs appear it's became a work for QA to find it. If this bugs slips away, well, it's definitly valid to blame QA for that. Yeah, QA can say "we found exactly that one, there is created task in tracker, from given date, with that exactly developer (or producer) who said it's not matter". It takes responsibility for that particular bug to somebody who closed the task, but it doesnt' take away responsibility for not creating the one. So, when we're thinking that developing process is structured nice, blaming the QA for such things is completly normal.
I know perfectly that you can't fish all glitches like that, maybe some kingdom creating crushs a game when performing with some obscure culture or with some trait, but skipping something like that is QA responsibility. That's why I was urging Paradox to hire more testers - because more testers find bugs more effective.

Why would someone continue to maintain and pay for 4 QA, while releasing closed beta for around 100-200 people would be better choice(imo) ?
https://forum.paradoxplaza.com/forum/index.php?threads/beta-testing-and-qa.827222/
 
Quality of reported bugs is the main reason. Beta testers are great, but they aren't paid to test and most of the time they'll test stuff that is fun. There are exceptions of course and some beta testers amaze us with the work they put in.
Paradox relied almost solely on beta testers up until V2 release (and then only as limited time contractors), EU4 was the first releases (as far as I remember with a in house team of QA). I'd say that quality has gone up since EU3 and HOI3 releases. QA isn't the only reason, but it's definitely one of the major.

Thanks for the answer.

Have you considered mixing things up? Having some QA people plus additional(don't know how many people gets to test the beta) close beta testers? I am sure some people would volunteer for that. I am asking because some of the Horse Lords exploits/cheesy tactics were easy to spot on by the average player.
 
Thanks for the answer.

Have you considered mixing things up? Having some QA people plus additional(don't know how many people gets to test the beta) close beta testers? I am sure some people would volunteer for that. I am asking because some of the Horse Lords exploits/cheesy tactics were easy to spot on by the average player.

They do exactly that. They actually asked for more beta-testers a few months ago.

Two things to add to aono's comments
1. Some 'obvious' bugs arent that obvious once you realise CK2's scope. Take the pagan reformation button going missing (on reload from memory). At the moment, there are broadly 7 groups with substantial different mechanics in the game (Catholic, Muslim, pagan/tribal, nomad, indian, merchant republic, ERE), 3 main starting dates (+ many more bookmarks), ways of life etc - you can't count on somebody playing x group, for long enough, and doing the necessary steps, and paying attention to x when there are a lot of other things that require attention too. Remember also that you dont see the bugs QA caught before it went live.
2. Its not a case of getting the 'final' product and wringing the bugs out of it for months. Bugs can be introduced in the process of fixing another bug (just check some of the notes from some of the hotfix patches for 'simple' fixes having unintended consequences), or balance or other changes don't work as intended.
 
  • 1
Reactions:
Quality of reported bugs is the main reason. Beta testers are great, but they aren't paid to test and most of the time they'll test stuff that is fun. There are exceptions of course and some beta testers amaze us with the work they put in.
Paradox relied almost solely on beta testers up until V2 release (and then only as limited time contractors), EU4 was the first releases (as far as I remember with a in house team of QA). I'd say that quality has gone up since EU3 and HOI3 releases. QA isn't the only reason, but it's definitely one of the major.

Whatever happened to that initiative around the time of WOL where the decision was made to release beta patches prior to expansions for the public to test and bugfix? That idea got canned pretty quickly it looks like, since we didn't see it for Horse Lords.
 
They do exactly that. They actually asked for more beta-testers a few months ago.

Two things to add to aono's comments
1. Some 'obvious' bugs arent that obvious once you realise CK2's scope. Take the pagan reformation button going missing (on reload from memory). At the moment, there are broadly 7 groups with substantial different mechanics in the game (Catholic, Muslim, pagan/tribal, nomad, indian, merchant republic, ERE), 3 main starting dates (+ many more bookmarks), ways of life etc - you can't count on somebody playing x group, for long enough, and doing the necessary steps, and paying attention to x when there are a lot of other things that require attention too. Remember also that you dont see the bugs QA caught before it went live.
2. Its not a case of getting the 'final' product and wringing the bugs out of it for months. Bugs can be introduced in the process of fixing another bug (just check some of the notes from some of the hotfix patches for 'simple' fixes having unintended consequences), or balance or other changes don't work as intended.

BZ1Ky0BIYAAYsH_.png
 
  • 10
  • 1
Reactions:
Whatever happened to that initiative around the time of WOL where the decision was made to release beta patches prior to expansions for the public to test and bugfix? That idea got canned pretty quickly it looks like, since we didn't see it for Horse Lords.

At the time is was stated that the reason why it was being done was because the DLC content was easily separable from the other changes (which were to the diplomatic UI).

HL wouldn't have been that easy to separate.

Can people just shut up about the Flanders issue,... I agree with your point, but I am getting sick of seeing it posted everywhere. It is about as annoying as the bloody china bandwagon!

I broadly agree and note that repeated off-topic posts tend to get regarded as spam.
 
Last edited:
  • 1
Reactions:
Can people just shut up about the Flanders issue,... I agree with your point, but I am getting sick of seeing it posted everywhere. It is about as annoying as the bloody china bandwagon!
...

I just want them to fix it... :(
 
  • 2
Reactions:
1. Some 'obvious' bugs arent that obvious once you realise CK2's scope. Take the pagan reformation button going missing (on reload from memory). At the moment, there are broadly 7 groups with substantial different mechanics in the game (Catholic, Muslim, pagan/tribal, nomad, indian, merchant republic, ERE), 3 main starting dates (+ many more bookmarks), ways of life etc - you can't count on somebody playing x group, for long enough, and doing the necessary steps, and paying attention to x when there are a lot of other things that require attention too. Remember also that you dont see the bugs QA caught before it went live.
Yup, I know. Once again - I know the drill. :)
I can't get how 4 testers can test scripted product at such scope, that's it. It takes efforts of 6 inner testers, outside publisher team who checked our tracker and closed beta test when we was making simple enough (and it became simpler when we did it to my cry) helicopter simulator, and I definitly know we missed something. Also 4 or 5 testers count normal for simple "Farm Frenzy"-type social network game (but one will sit on monetization all the time and another on synchronization tbh).

I love how people on this forum think they can do Paradox's job better than them.
Not every part.
I'm very, very, very auful coder, so I've never think I can say something about actual developing (I mean code writing). Also I've never been a publisher, so I've never say something about manager or producer work. It's just not my area of expertise. I have an opinion how a game can be done better (some mechanics or something), but I understand clearly that it can take too many effort to put that ideas into game, so I rarely advice something like that.
But I started QA work in 2002, so I believe I earned a right to have an opinion here.
 
Thanks for the answer.

Have you considered mixing things up? Having some QA people plus additional(don't know how many people gets to test the beta) close beta testers? I am sure some people would volunteer for that. I am asking because some of the Horse Lords exploits/cheesy tactics were easy to spot on by the average player.

We have a closed beta for CK2 already and some 30-50 testers (I'm not up to date with the exact number). I think they're listed in credits for our expansions. Every now and then we hold open applications for becoming a beta tester. Often we select people who've been active in the bug forums.
 
  • 3
  • 1
Reactions:
I love how people on this forum think they can do Paradox's job better than them.

About coding? Depends. You know where people work in real life? No.

About gameplay? Depends. You know how many games people played? No.

Many modders did better job overall than Paradox on their respective games. I´ve started playing Pdox games for real... with EU3 AND Magna Mundi mod circa 2008. Many concepts of which were seen in some form later in EU4.

Many people only played Victoria 2 with Naselus´ mod.

I could go on and on. Being dismissive of input is as wrong as listening to it always. The problem is filtering information.
 
  • 7
  • 3
Reactions: