CK3 Dev Diary #35 - User Testing Before Release

[holyvigil]

I feel like the decision to make 3d models will prove to be a pivotal point in CKs overall life time. Paradox has chosen to focus on the RP aspects of CK in a way which has never been viscerally done before.

 

[mteir]

For all the complaints about the portraits, let's also remember what CK1's portraits looked like. If you're thinking the early CK3 portraits are cartoonish or something out of a mobile game, you haven't seen anything.

The prequel series had the best portraits.

Spoiler: Svea Rike

 

[Nerewar90]

For all the complaints about the portraits, let's also remember what CK1's portraits looked like. If you're thinking the early CK3 portraits are cartoonish or something out of a mobile game, you haven't seen anything.

Yes, but it was released in 2004 and today is 2020. I'm really glad they removed those early cartoonish portraits.

 

[Weyird]

From everything I've seen in previous dev diaries, the UI / UX for CK3 seems really well done. Everything is looks pretty clear. Which I can't say for all games. Also, not meaning to hate on Imperator, but was User Research even done for that game? Primarily because the UI is a mess.

 

[Riekopo]

And good thing we got some feedback, because the style turned out to be way too cartoony for most people! It might have brought our colleagues a lot of work, but I’m glad we discovered this way before release day…

I'm sorry. You actually thought that cartoon looked good? Wat???

 

[Sphem]

The million dollar question: did any of the user testers for Imperator complain about mana? Of course the sample sizes are limited but I feel like discovering a fundamentally annoying game concept ahead of time is very much in the scope of user testing.

I think mana was just getting old in pds games and imperator was the straw that broke the camels back. Its not as much a game concept as it was thought to be a necessity in these games. These grand strategy games originated from the idea of a board game in real time and board games very much use the mana as a tool. In reality they had to expand beyond mana, and I think ck3’s focus on roleplay has helped it tremendously in this area.

 

[Masternachos]

Well I know who I'M marrying!

...Because she's a modern-day person and therefore can't possibly be one of my ancient long-lost ancestors, which makes things weird if you think about it that way.

 

[Sugary]

Look at that learning...one way or another she'll be working at my court

 

[Ivashanko]

When User Research are asked to evaluate a feature, we look at the game from a user perspective by trying to test and gather feedback on things like how well players understand the game, if any menus are confusing, if players are having fun, etc. This can be done in a variety of ways, but for Crusader Kings III we mostly used playtests where we let 2-15 players that match a specific profile (in this case - PDS strategy players) into our lab to play an unfinished version of the game while watching and recording them and then we brought video clips and player feedback to the dev team.

Woah, that seems pretty abstract!

...does it?

 

[Heiliges Thüringens Reich]

I'm really glad your job exists. Companies listening to player feedback is paramount!

Spoiler

PS: EU4 desperately need someone like you in their team.

 

[uleslaw]

That was an interesting reading, thanks. What is your background? Are you a graduate psychologist?

 

[Olden Weiss]

That was an interesting reading, thanks. What is your background? Are you a graduate psychologist?

I'd imagine it's less that and more statistician. It sounds to me like this kind of job requires a great deal of knowledge in what is and isn't constructive to pass on to development. After all, saddling the programmers with every little comment about how ugly a character's baby ended up would just bog down development. I figure a job like this requires a keen understanding of what can and can't be used for good. A bit like an intelligence officer, only with far less lives on the line.

 

[MarkFranz]

Let us be fair for a moment, gentlemen.

As bad as the original portraits looked, let's remember that this was concept art and refinements certainly would have been made. I absolutely agree that we dodged the arrow by way of the change in direction, but let's not be too hard on the art team. Concepts rarely look as gritty as their final form, even when the art direction isn't changed. As an outdated example...

Spoiler: image

View attachment 603612

But unlike portaits showed in this DD, this concept and ngame model are both great.

 

[Syroc]

So, when you guys are testing gameplay, do you guys mostly do it multiplayer, singleplayer or an even mix of the two? Because thinking back to how Stellaris and Imperator (and even the base CKII, but there were also RPG, RNG and silly bug shenanigans that prolonged my interest) released, those experiences got really old really fast in singleplayer, something that I suspect wouldn't be readily apparent if your playtesters were all having a good time together in a campaign.

 

[Ezumiyr]

I wonder how they determine this. Offline play doesn't count to steam registered time afaik. And people might have the game running for ages idle. Or they might be testers of the previous game. Game development often use different depots as time goes by.

I mean, even if you take that into account, players with 200-800 hours according to Steam all probably know how to play CK2.
Nobody lets the game ideal for 180 hours and only actually play for 20.
And if it's people who actually played for more than 1500 hours offline, well, it doesn't really matter at this point.

In my experience, people are asked how much time they play several video games anyway (so if you register to playtest an MMO for example, they ask you how much you played WoW and several others). Since it's self-reported, it's rather subjective. What matters is the order of magnitude, from total newbie to deep addiction.

 

[Jayavarman]

For all the complaints about the portraits, let's also remember what CK1's portraits looked like. If you're thinking the early CK3 portraits are cartoonish or something out of a mobile game, you haven't seen anything.

I and the playerbase loved CK1 portraits. They aesthetically fit with the UI and game context as well as were a revolutionary feature for Paradox/strategy games at the time.

 

[Varus90]

I and the playerbase loved CK1 portraits. They aesthetically fit with the UI and game context as well as were a revolutionary feature for Paradox/strategy games at the time.

Yeah, but too bad there were only like five of them per culture, so you, your best friend, and your worst enemy could all look the same.

 

[March_Cat]

Thank You, Hannah! You have a very interesting job! Can you tell us about how you started doing this?

 

[Dayvit78]

Hello everyone!

My name is Hanna and I work in a growing field of game development called User Research! I’m here to tell you a little about the job itself and also how me and my team have worked with user testing for Crusader Kings III.


So if you run these tests why aren’t the games perfect and amazing then?
Oh, I ask myself the same thing every day… But the truth is, game development is complicated and sometimes there’s just no time, manpower or money to address the issues we’ve discovered. We also share our results with community management and marketing to give them a heads up on the questions and concerns they can expect on release.

As someone who works in Market Research (FMCG industry), I appreciate your sharing! As to the question above, you answered it. Market research may have the answers, but they don't have all the power - we just present our findings to the actual decision makers. They can agree or disagree, and follow our recommendations or not. So please don't immediately put the blame on the researchers!

 

[BrotherJonathan]

Keep up the good work!