• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

CK3 Dev Diary #35 - User Testing Before Release

Hello everyone!

My name is Hanna and I work in a growing field of game development called User Research! I’m here to tell you a little about the job itself and also how me and my team have worked with user testing for Crusader Kings III.

Ah, user testing, I get it - you’re QA/Quality Assurance!
Oh, how many times I hear that! But no, I am not. Very, very simplified it’s kind of like this: When QA evaluate a feature they report on it in a yes/no kind of way - Does it work? Yes/No, so here’s a bug report.

When User Research are asked to evaluate a feature, we look at the game from a user perspective by trying to test and gather feedback on things like how well players understand the game, if any menus are confusing, if players are having fun, etc. This can be done in a variety of ways, but for Crusader Kings III we mostly used playtests where we let 2-15 players that match a specific profile (in this case - PDS strategy players) into our lab to play an unfinished version of the game while watching and recording them and then we brought video clips and player feedback to the dev team.

Woah, that seems pretty abstract! How can you make sure that you’re getting these things completely right and accurate?
You’re right, it is tough - We do this by using a lot of methods used in social science to be able to verify our findings. Most of the time we’ll use a combination of observation (meaning we watch the player playing and take notes on what they say/do), one or several surveys and then an interview. If we can see several people struggle with a tutorial step, chances are they won’t be alone in having that issue, and having them describe how it felt both verbally and in text let’s us pinpoint what’s wrong.

But of course humans are complicated, and it’s important to remember that while we use scientific methods our job is not to produce scientific results. Instead, our job is giving our colleagues a heads up on how to tweak things by providing a basis for discussion and re-design.

And this is relevant to Crusader Kings III because…?
Well, I think having User Research done is good for any kind of interactive medium when it comes to evaluating the ideas and designs put into the game. Crusader Kings III is also an interesting challenge from a user design perspective - How do we keep the complexity hardcore players want while still making sure new players can discover the awesome feeling of inheriting a title because your nephew had an “accident”? If we don’t listen to what actual players say and think, we risk making mistakes and finding out about it when it’s too late to change anything.

Enough about Using Research or whatever, what did you do for Crusader Kings III?
As you’ve likely seen we’re doing a lot of new things in this game compared to both other PDS titles and Crusader Kings II, so this means that having player feedback early is a good idea to avoid going ahead with something that turns out to be a mistake.

For example, one of the first things we did for Crusader Kings III was actually done internally - the dev team were making early designs for characters and User Research helped them out by sending out a survey to all of our colleagues to gather feedback. At the time, characters looked like this:

Early concept arts.jpg


And good thing we got some feedback, because the style turned out to be way too cartoony for most people! It might have brought our colleagues a lot of work, but I’m glad we discovered this way before release day…

Throughout the development of the game, we’ve also run bi-monthly 1 hour playtests with 2-4 players. These tests are often done in close collaboration with the UX designers and are very focused on us seeing how well players are able to finish basic gameplay actions like getting married, starting and finishing a war, setting up a scheme and things like that. Thanks to these tests, several changes have been made to menus that have proven difficult for players to navigate like the Council, Marriage and Building menus and the Siege/War overviews.

But my favourite part of my work is to actually see people play and experience the game during longer playtests. For Crusader Kings III the most important playtest was when it was time to evaluate the tutorial and systems for in-game help. For this test we had 12 participants, 6 of them with 200-800h of CK2 playtime and 6 of them “newbies”, try out the tutorial and then play the game for 2 hours using what they’d learned.

And you know what? Most of them managed just fine, which isn’t really something I’m used to seeing when it comes to Paradox onboarding. New players loved that the tutorial used visual cues to guide them through menus and the suggestions helped them with an issue that many PDS titles have - What the heck do I do next? “Old” players found the UI took some time to get used to, but they also found the suggestion systems useful as checklists on what to do next.

But of course, nothing is perfect - and thanks to the player feedback we could also pinpoint areas where more work was needed for both new and old players, especially regarding the UI. But throughout the playtests we’ve done I’ve been happy to see the game go from incomprehensible to approachable, both for new and returning Crusader Kings-fans.

So if you run these tests why aren’t the games perfect and amazing then?
Oh, I ask myself the same thing every day… But the truth is, game development is complicated and sometimes there’s just no time, manpower or money to address the issues we’ve discovered. We also share our results with community management and marketing to give them a heads up on the questions and concerns they can expect on release.

Has anything changed due to Corona?
Oh absolutely! The biggest change for my team of researchers is that we don’t have access to our beautiful lab at the Stockholm office. The lab is geared up with computer stations where you can prep game installation and recording setup beforehand. This makes setting up studies a lot easier for us and it’s also a good way of preventing leaks.
Crusader Kings III has been lucky in that we had concluded most of the lab activities we’d planned before the release, but for other projects we’ve done studies remotely meaning that we let players play the game on their own computers and do all interviews via video chats. While we’re still trying to work things out, we’re also discovering a lot of benefits from this method of working - the biggest one being that we could potentially run playtests with people from all over the world!

What will you do after the release?
For Crusader Kings III, I will start working on gathering and reporting on feedback following the release - which means that anything you write on Steam or other pages, I’ll go over and try to summarize topics so we know what to improve on next. Then, it’s time to plan new playtests for patches and DLCs!

I hope you’ve enjoyed reading about a lesser known field in game development and that you’ll look forward to the upcoming release of a game that I’ve absolutely loved working with. I will be on vacation when this is posted, but I’ll be happy to answer any questions about User Research when I’m back again on August 1st. Oh, and please be nice to my vanity character if you encounter her - she’s very, very weak.

Vanity Character.jpg
 
  • 103
  • 74Like
  • 11
  • 4Love
  • 4
Reactions:
I feel like the decision to make 3d models will prove to be a pivotal point in CKs overall life time. Paradox has chosen to focus on the RP aspects of CK in a way which has never been viscerally done before.
 
  • 27
  • 1Like
Reactions:
For all the complaints about the portraits, let's also remember what CK1's portraits looked like. If you're thinking the early CK3 portraits are cartoonish or something out of a mobile game, you haven't seen anything.
The prequel series had the best portraits.
75934-svea-rike-windows-screenshot-scenario-start-screen.gif
 
  • 3
  • 1Haha
Reactions:
For all the complaints about the portraits, let's also remember what CK1's portraits looked like. If you're thinking the early CK3 portraits are cartoonish or something out of a mobile game, you haven't seen anything.

UO0019.jpg

Yes, but it was released in 2004 and today is 2020. I'm really glad they removed those early cartoonish portraits.
 
  • 12
  • 3Like
  • 1
Reactions:
From everything I've seen in previous dev diaries, the UI / UX for CK3 seems really well done. Everything is looks pretty clear. Which I can't say for all games. Also, not meaning to hate on Imperator, but was User Research even done for that game? Primarily because the UI is a mess.
 
  • 9
  • 1Like
  • 1
  • 1
Reactions:
And good thing we got some feedback, because the style turned out to be way too cartoony for most people! It might have brought our colleagues a lot of work, but I’m glad we discovered this way before release day…

I'm sorry. You actually thought that cartoon looked good? Wat???
 
  • 6Haha
  • 3Like
  • 3
  • 1
Reactions:
The million dollar question: did any of the user testers for Imperator complain about mana? Of course the sample sizes are limited but I feel like discovering a fundamentally annoying game concept ahead of time is very much in the scope of user testing.
I think mana was just getting old in pds games and imperator was the straw that broke the camels back. Its not as much a game concept as it was thought to be a necessity in these games. These grand strategy games originated from the idea of a board game in real time and board games very much use the mana as a tool. In reality they had to expand beyond mana, and I think ck3’s focus on roleplay has helped it tremendously in this area.
 
  • 10
  • 2Like
  • 1
Reactions:
Well I know who I'M marrying!

...Because she's a modern-day person and therefore can't possibly be one of my ancient long-lost ancestors, which makes things weird if you think about it that way.
 
  • 4Haha
  • 1
  • 1
Reactions:
When User Research are asked to evaluate a feature, we look at the game from a user perspective by trying to test and gather feedback on things like how well players understand the game, if any menus are confusing, if players are having fun, etc. This can be done in a variety of ways, but for Crusader Kings III we mostly used playtests where we let 2-15 players that match a specific profile (in this case - PDS strategy players) into our lab to play an unfinished version of the game while watching and recording them and then we brought video clips and player feedback to the dev team.

Woah, that seems pretty abstract!

...does it?
 
  • 5
  • 1Haha
  • 1
Reactions:
I'm really glad your job exists. Companies listening to player feedback is paramount!

PS: EU4 desperately need someone like you in their team. ;)
 
  • 6
  • 2Like
Reactions:
That was an interesting reading, thanks. What is your background? Are you a graduate psychologist?
 
  • 2Like
  • 1
Reactions:
That was an interesting reading, thanks. What is your background? Are you a graduate psychologist?

I'd imagine it's less that and more statistician. It sounds to me like this kind of job requires a great deal of knowledge in what is and isn't constructive to pass on to development. After all, saddling the programmers with every little comment about how ugly a character's baby ended up would just bog down development. I figure a job like this requires a keen understanding of what can and can't be used for good. A bit like an intelligence officer, only with far less lives on the line.
 
  • 4
  • 1Like
  • 1
Reactions:
Let us be fair for a moment, gentlemen.

As bad as the original portraits looked, let's remember that this was concept art and refinements certainly would have been made. I absolutely agree that we dodged the arrow by way of the change in direction, but let's not be too hard on the art team. Concepts rarely look as gritty as their final form, even when the art direction isn't changed. As an outdated example...
But unlike portaits showed in this DD, this concept and ngame model are both great.
 
  • 6
  • 1
Reactions:
So, when you guys are testing gameplay, do you guys mostly do it multiplayer, singleplayer or an even mix of the two? Because thinking back to how Stellaris and Imperator (and even the base CKII, but there were also RPG, RNG and silly bug shenanigans that prolonged my interest) released, those experiences got really old really fast in singleplayer, something that I suspect wouldn't be readily apparent if your playtesters were all having a good time together in a campaign.
 
  • 1Like
  • 1
Reactions:
I wonder how they determine this. Offline play doesn't count to steam registered time afaik. And people might have the game running for ages idle. Or they might be testers of the previous game. Game development often use different depots as time goes by.
I mean, even if you take that into account, players with 200-800 hours according to Steam all probably know how to play CK2.
Nobody lets the game ideal for 180 hours and only actually play for 20.
And if it's people who actually played for more than 1500 hours offline, well, it doesn't really matter at this point.

In my experience, people are asked how much time they play several video games anyway (so if you register to playtest an MMO for example, they ask you how much you played WoW and several others). Since it's self-reported, it's rather subjective. What matters is the order of magnitude, from total newbie to deep addiction.
 
  • 1Like
  • 1
  • 1
Reactions:
For all the complaints about the portraits, let's also remember what CK1's portraits looked like. If you're thinking the early CK3 portraits are cartoonish or something out of a mobile game, you haven't seen anything.

UO0019.jpg
I and the playerbase loved CK1 portraits. They aesthetically fit with the UI and game context as well as were a revolutionary feature for Paradox/strategy games at the time.
 
  • 7
  • 3
  • 1Like
Reactions:
I and the playerbase loved CK1 portraits. They aesthetically fit with the UI and game context as well as were a revolutionary feature for Paradox/strategy games at the time.
Yeah, but too bad there were only like five of them per culture, so you, your best friend, and your worst enemy could all look the same.
 
  • 1Haha
Reactions:
Hello everyone!

My name is Hanna and I work in a growing field of game development called User Research! I’m here to tell you a little about the job itself and also how me and my team have worked with user testing for Crusader Kings III.


So if you run these tests why aren’t the games perfect and amazing then?
Oh, I ask myself the same thing every day… But the truth is, game development is complicated and sometimes there’s just no time, manpower or money to address the issues we’ve discovered. We also share our results with community management and marketing to give them a heads up on the questions and concerns they can expect on release.

As someone who works in Market Research (FMCG industry), I appreciate your sharing! As to the question above, you answered it. Market research may have the answers, but they don't have all the power - we just present our findings to the actual decision makers. They can agree or disagree, and follow our recommendations or not. So please don't immediately put the blame on the researchers!
 
  • 7Like
  • 2
  • 1
Reactions:
Keep up the good work!