CK3 Dev Diary #35 - User Testing Before Release

CK3 Dev Diary #35 - User Testing Before Release

  • Crusader Kings III Available Now!

    The realm rejoices as Paradox Interactive announces the launch of Crusader Kings III, the latest entry in the publisher’s grand strategy role-playing game franchise. Advisors may now jockey for positions of influence and adversaries should save their schemes for another day, because on this day Crusader Kings III can be purchased on Steam, the Paradox Store, and other major online retailers.


    Real Strategy Requires Cunning

Olden Weiss

Major
Jul 11, 2020
753
3.319
If you read the entire dev diary, you'll see that they perform many tests, and one of them at least involved total newbies.
This may well have been what led to embedded tooltips and the tutorial that so many first-time players are raving about in the professional review scene. We never really think about how important thankless work like that done by Hanna's team is until we stop to think about it. As
ashbery76 said, the portraits alone may have caused a war if they weren't caught!

Glad we got this dev diary. It's important we appreciate the fact that not only are the devs asking players what they like and dislike about the game, but that there are professionals who are able to turn that feedback into actionable data.
 
  • 24
  • 3Like
Reactions:

Nikolai

Basileus Romaion
71 Badges
Jun 17, 2001
19.557
1.735
  • Europa Universalis IV: Pre-order
  • Europa Universalis: Rome
  • Semper Fi
  • Sengoku
  • Supreme Ruler: Cold War
  • Victoria 2
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Rome: Vae Victis
  • 500k Club
  • Cities: Skylines
  • Crusader Kings II: Holy Knight (pre-order)
  • Europa Universalis III: Collection
  • Victoria: Revolutions
  • Europa Universalis: Rome Collectors Edition
  • Mount & Blade: Warband
  • Crusader Kings II: Way of Life
  • Pillars of Eternity
  • Europa Universalis IV: Common Sense
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Hearts of Iron IV Sign-up
  • Stellaris Sign-up
  • Hearts of Iron IV: Cadet
  • Stellaris: Lithoids
  • Europa Universalis III Complete
  • Hearts of Iron II: Armageddon
  • Crusader Kings II
  • Crusader Kings II: Charlemagne
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Sword of Islam
  • Darkest Hour
  • Deus Vult
  • Europa Universalis III
  • Europa Universalis III: Chronicles
  • Divine Wind
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Call to arms event
  • For The Glory
  • For the Motherland
  • Hearts of Iron III
  • Heir to the Throne
  • Europa Universalis III Complete
  • Magicka
  • March of the Eagles
How do you gather the right people for remote user testing? I bet that can't be easy! I mean, you can't use only forum members I guess, even if they would be the easier path. We're kind of the most passionate and dedicated fans on guess, but you'd need some more diverse base, right? And if you ask for signups, you'd again get the ones following your twitter, fb etc accounts I guess? Then comes reliability, both for nda and actually showing up, but also having a good system to record reactions. Do you use people with webcams on? And do you use people several times to record changes over time with the same people for reference?

Sorry for the many questions, this is intriguing. I have much experience as an alpha and beta tester for many games, both yours and other companies over well over a decade now. But this is an unknown area that intrigues me.
 
  • 7Like
  • 1
Reactions:

Olden Weiss

Major
Jul 11, 2020
753
3.319
  • 8Haha
Reactions:

Fallender

Second Lieutenant
81 Badges
May 9, 2012
164
84
  • Crusader Kings II
  • Cities: Skylines
  • Stellaris: Humanoids Species Pack
  • Imperator: Rome - Magna Graecia
  • Stellaris: Apocalypse
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Stellaris: Distant Stars
  • Shadowrun Returns
  • Shadowrun: Dragonfall
  • Stellaris: Megacorp
  • Crusader Kings II: Holy Fury
  • Imperator: Rome Deluxe Edition
  • Prison Architect
  • Imperator: Rome Sign Up
  • Stellaris: Ancient Relics
  • Age of Wonders: Planetfall
  • Age of Wonders: Planetfall Deluxe edition
  • Age of Wonders: Planetfall Premium edition
  • Age of Wonders: Planetfall Season pass
  • Crusader Kings III Referal
  • Stellaris: Lithoids
  • Age of Wonders: Planetfall - Revelations
  • Crusader Kings II: Conclave
  • Teleglitch: Die More Edition
  • War of the Roses
  • 500k Club
  • Europa Universalis IV: Pre-order
  • Magicka: Wizard Wars Founder Wizard
  • Mount & Blade: Warband
  • Mount & Blade: With Fire and Sword
  • Crusader Kings II: Way of Life
  • Magicka 2
  • Cities: Skylines - After Dark
  • Crusader Kings II: Jade Dragon
  • Stellaris
  • Stellaris: Galaxy Edition
  • Hearts of Iron IV: Cadet
  • Hearts of Iron IV: Colonel
  • Crusader Kings II: Reapers Due
  • Tyranny: Archon Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Crusader Kings II: Monks and Mystics
  • Age of Wonders III
For this test we had 12 participants, 6 of them with 200-800h of CK2 playtime and 6 of them “newbies”, try out the tutorial and then play the game for 2 hours using what they’d learned
200-800? So you mean only newbies?
 
  • 40Haha
  • 7
  • 1
Reactions:

Nikolai

Basileus Romaion
71 Badges
Jun 17, 2001
19.557
1.735
  • Europa Universalis IV: Pre-order
  • Europa Universalis: Rome
  • Semper Fi
  • Sengoku
  • Supreme Ruler: Cold War
  • Victoria 2
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Rome: Vae Victis
  • 500k Club
  • Cities: Skylines
  • Crusader Kings II: Holy Knight (pre-order)
  • Europa Universalis III: Collection
  • Victoria: Revolutions
  • Europa Universalis: Rome Collectors Edition
  • Mount & Blade: Warband
  • Crusader Kings II: Way of Life
  • Pillars of Eternity
  • Europa Universalis IV: Common Sense
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Hearts of Iron IV Sign-up
  • Stellaris Sign-up
  • Hearts of Iron IV: Cadet
  • Stellaris: Lithoids
  • Europa Universalis III Complete
  • Hearts of Iron II: Armageddon
  • Crusader Kings II
  • Crusader Kings II: Charlemagne
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Sword of Islam
  • Darkest Hour
  • Deus Vult
  • Europa Universalis III
  • Europa Universalis III: Chronicles
  • Divine Wind
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Call to arms event
  • For The Glory
  • For the Motherland
  • Hearts of Iron III
  • Heir to the Throne
  • Europa Universalis III Complete
  • Magicka
  • March of the Eagles
200-800? So you mean only newbies?
I wonder how they determine this. Offline play doesn't count to steam registered time afaik. And people might have the game running for ages idle. Or they might be testers of the previous game. Game development often use different depots as time goes by.
 

Keizer Harm

Swamp German
39 Badges
Jan 28, 2013
2.440
2.602
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings III
The million dollar question: did any of the user testers for Imperator complain about mana? Of course the sample sizes are limited but I feel like discovering a fundamentally annoying game concept ahead of time is very much in the scope of user testing.
 
  • 17
Reactions:

Cyhort

First Lieutenant
On Probation
63 Badges
Jan 22, 2009
289
905
  • Deus Vult
  • Rome Gold
  • Heir to the Throne
  • Sengoku
  • March of the Eagles
  • East India Company
  • Commander: Conquest of the Americas
  • Europa Universalis IV: Rights of Man
  • Europa Universalis IV: El Dorado
  • Mount & Blade: Warband
  • Crusader Kings II: Way of Life
  • Europa Universalis IV: Common Sense
  • Crusader Kings II: Horse Lords
  • Crusader Kings II: Conclave
  • Stellaris
  • Hearts of Iron IV: Cadet
  • Crusader Kings II: Reapers Due
  • Imperator: Rome - Magna Graecia
  • Stellaris: Leviathans Story Pack
  • Hearts of Iron IV: Together for Victory
  • Crusader Kings II: Monks and Mystics
  • Stellaris: Apocalypse
  • Europa Universalis IV: Mandate of Heaven
  • Hearts of Iron IV: Death or Dishonor
  • Stellaris: Synthetic Dawn
  • Age of Wonders III
  • Hearts of Iron IV: Expansion Pass
  • Stellaris: Humanoids Species Pack
  • Europa Universalis III Complete
  • Shadowrun Returns
  • Shadowrun: Dragonfall
  • Crusader Kings II: Holy Fury
  • Europa Universalis IV: Dharma
  • Stellaris: Distant Stars
  • Shadowrun: Hong Kong
  • Imperator: Rome Deluxe Edition
  • Europa Universalis III Complete
  • Supreme Ruler 2020
  • Victoria 2
  • Victoria 2: A House Divided
  • 500k Club
  • Crusader Kings II: Holy Knight (pre-order)
  • Victoria 2: Heart of Darkness
  • Europa Universalis IV: Res Publica
  • Europa Universalis IV: Pre-order
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Art of War
  • Europa Universalis IV: Third Rome
  • Crusader Kings II: Jade Dragon
I'd like to add my horror at those original portraits.
 
  • 9
  • 3Haha
  • 1Like
Reactions:

wilcoxchar

Field Marshal
93 Badges
Nov 15, 2004
3.113
3.297
  • Europa Universalis III: Collection
  • Victoria: Revolutions
  • Europa Universalis: Rome
  • Semper Fi
  • Sengoku
  • Victoria 2
  • Victoria 2: Heart of Darkness
  • Rome: Vae Victis
  • 200k Club
  • 500k Club
  • Cities: Skylines
  • Cities: Skylines Deluxe Edition
  • Crusader Kings II: Holy Knight (pre-order)
  • Europa Universalis IV: Res Publica
  • Europa Universalis IV: El Dorado
  • Europa Universalis IV: Pre-order
  • Europa Universalis: Rome Collectors Edition
  • Mount & Blade: Warband
  • Mount & Blade: With Fire and Sword
  • Crusader Kings II: Way of Life
  • Europa Universalis IV: Common Sense
  • Crusader Kings II: Horse Lords
  • Cities: Skylines - After Dark
  • Europa Universalis IV: Cossacks
  • Crusader Kings II: Conclave
  • Crusader Kings III
  • Europa Universalis III
  • Hearts of Iron Anthology
  • Crusader Kings II
  • Crusader Kings II: Charlemagne
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: Rajas of India
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Sword of Islam
  • Deus Vult
  • Diplomacy
  • Divine Wind
  • Europa Universalis IV: Art of War
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis IV: Call to arms event
  • For The Glory
  • Hearts of Iron III
  • Heir to the Throne
  • Europa Universalis III Complete
  • March of the Eagles
Thank you for the explanation of user research! Making the game accessible to a variety of players is very important and often far too underappreciated part of game design. The kind of playtesting required to make sure elements of the game aren't confusing or frustrating to learn how to use requires a lot more time and effort than some people might think and I'm glad Paradox is taking the time and effort to make sure new and experienced players alike can get into CK3.
 
  • 1
  • 1Like
Reactions:

grandad1982

Field Marshal
100 Badges
May 1, 2012
3.439
4.320
  • Sengoku
  • Hearts of Iron III: Their Finest Hour
  • Heir to the Throne
  • Stellaris
  • Knights of Pen and Paper +1 Edition
  • March of the Eagles
  • Europa Universalis IV: Third Rome
  • Europa Universalis IV: Res Publica
  • Victoria: Revolutions
  • Semper Fi
  • Cities in Motion 2
  • Sword of the Stars
  • Cities: Skylines - Snowfall
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Europa Universalis IV: Cossacks
  • Cities: Skylines - After Dark
  • Stellaris - Path to Destruction bundle
  • Crusader Kings II: Jade Dragon
  • Europa Universalis IV: Pre-order
  • For the Motherland
  • Crusader Kings II
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: Rajas of India
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Sword of Islam
  • Europa Universalis III
  • Europa Universalis III Complete
  • Divine Wind
  • Europa Universalis IV: Art of War
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis IV: Call to arms event
  • Hearts of Iron III
  • Crusader Kings II: Conclave
  • Hearts of Iron IV Sign-up
  • Crusader Kings II: Horse Lords
  • Europa Universalis IV: Common Sense
  • Pillars of Eternity
  • Crusader Kings II: Way of Life
  • 500k Club
  • Warlock: Master of the Arcane
  • Victoria 2
  • Teleglitch: Die More Edition
  • Europa Universalis III Complete
  • Crusader Kings III
Probably because a great many people have mobile devices and use them regularly. I don't expect PDS will be going there anytime soon, though. Their games fit a popular niche, and they do so because of depth that would be incredibly difficult to capture with mobile hardware.
Say hello to Stellaris: Galaxy Command. Admittedly not pds but pdx.
 
  • 7
Reactions:

Rockphed

Uncultured Barbarian
16 Badges
Jan 31, 2017
969
47
  • Crusader Kings II
  • Crusader Kings II: Way of Life
  • Crusader Kings II: Horse Lords
  • Crusader Kings II: Reapers Due
  • Crusader Kings II: Monks and Mystics
  • Crusader Kings II: Jade Dragon
  • Shadowrun Returns
  • Crusader Kings II: Holy Fury
  • Crusader Kings II: Charlemagne
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: Rajas of India
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Sword of Islam
Ironing out bits and bobbles, you will know soon enough ;)
We should make a "Fete Rod" picture for when we like what you do. I look forward to watching you play CKIII with the same gusto you used when narrating the Imperator dev clash.

Those old portraits look like they'd be right at home in some tacky, outsourced free-to-play mobile social spin-off. Crusader Friends™ perhaps?
I hope that's not giving any ideas
Now I want to figure out what Crusader Kings would look like as a mobile game. It would end up being some sort of pyramid scheme where you use your resources to make new dynasty members, marry them into other people's courts, and then use them to plot to overthrow the person who got you in to the game.

is the character creator in the game?
In the thread last week it was mentioned that there is a console command to open the character editor. You use the sliders to change appearance and then can copy out the DNA string into a mod file.
 
  • 2
  • 1Haha
Reactions:

Olden Weiss

Major
Jul 11, 2020
753
3.319
  • 1
Reactions:

Vanhal

Lt. General
52 Badges
Nov 19, 2009
1.373
2.086
  • Crusader Kings III
  • Crusader Kings III Referal
  • 500k Club
  • Stellaris Sign-up
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Synthetic Dawn
  • Stellaris: Leviathans Story Pack
  • Stellaris - Path to Destruction bundle
  • Stellaris: Megacorp
  • Stellaris: Humanoids Species Pack
  • Stellaris: Distant Stars
  • Stellaris: Apocalypse
  • Stellaris: Ancient Relics
  • Stellaris: Lithoids
  • BATTLETECH
  • BATTLETECH - Digital Deluxe Edition
  • BATTLETECH: Season pass
  • BATTLETECH: Flashpoint
  • BATTLETECH: Heavy Metal
  • Crusader Kings II
  • Crusader Kings II: Sword of Islam
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Republic
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Rajas of India
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Conclave
  • Crusader Kings II: Charlemagne
  • Crusader Kings II: Horse Lords
  • Crusader Kings II: Reapers Due
  • Crusader Kings II: Way of Life
  • Crusader Kings II: Monks and Mystics
  • Crusader Kings II: Jade Dragon
  • Crusader Kings II: Holy Fury
  • Imperator: Rome Sign Up
  • Imperator: Rome
  • Imperator: Rome Deluxe Edition
  • Imperator: Rome - Magna Graecia
  • Age of Wonders III
  • Age of Wonders: Planetfall Sign Up
  • Age of Wonders: Planetfall
  • Age of Wonders: Planetfall - Revelations
  • Sengoku
  • Victoria 2
  • Victoria 2: A House Divided
I will probably say what everyone here said already, but i am so incredibly glad you didn't go the awful, awful way of Civilization 6, where the leader portraits are so incredibly unimmersive and horrible that the only diplomacy possible with most of them is "your head would look better on the top of a pole".
 
  • 8Haha
  • 3
  • 1
Reactions:

Olden Weiss

Major
Jul 11, 2020
753
3.319
I will probably say what everyone here said already, but i am so incredibly glad you didn't go the awful, awful way of Civilization 6, where the leader portraits are so incredibly unimmersive and horrible that the only diplomacy possible with most of them is "your head would look better on the top of a pole".
I actually rather like Civ 6's portraits. They're charming, in their way. However I agree that the look objectively would not have worked for CK3. Civ 6 is, at the end of the day, a game. Its entire format reminds you of that regularly, and thus it can afford to have a very deliberate stylistic look. CK3, on the other hand, is as much roleplaying as it is a game and is heavily grounded in history, so such a look doesn't do its setting justice.
 
  • 22
  • 5
  • 1
Reactions:

Olden Weiss

Major
Jul 11, 2020
753
3.319
Let us be fair for a moment, gentlemen.

As bad as the original portraits looked, let's remember that this was concept art and refinements certainly would have been made. I absolutely agree that we dodged the arrow by way of the change in direction, but let's not be too hard on the art team. Concepts rarely look as gritty as their final form, even when the art direction isn't changed. As an outdated example...

1595949530175.png
 
  • 11
  • 2Like
  • 2
Reactions:

wilcoxchar

Field Marshal
93 Badges
Nov 15, 2004
3.113
3.297
  • Europa Universalis III: Collection
  • Victoria: Revolutions
  • Europa Universalis: Rome
  • Semper Fi
  • Sengoku
  • Victoria 2
  • Victoria 2: Heart of Darkness
  • Rome: Vae Victis
  • 200k Club
  • 500k Club
  • Cities: Skylines
  • Cities: Skylines Deluxe Edition
  • Crusader Kings II: Holy Knight (pre-order)
  • Europa Universalis IV: Res Publica
  • Europa Universalis IV: El Dorado
  • Europa Universalis IV: Pre-order
  • Europa Universalis: Rome Collectors Edition
  • Mount & Blade: Warband
  • Mount & Blade: With Fire and Sword
  • Crusader Kings II: Way of Life
  • Europa Universalis IV: Common Sense
  • Crusader Kings II: Horse Lords
  • Cities: Skylines - After Dark
  • Europa Universalis IV: Cossacks
  • Crusader Kings II: Conclave
  • Crusader Kings III
  • Europa Universalis III
  • Hearts of Iron Anthology
  • Crusader Kings II
  • Crusader Kings II: Charlemagne
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: Rajas of India
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Sword of Islam
  • Deus Vult
  • Diplomacy
  • Divine Wind
  • Europa Universalis IV: Art of War
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis IV: Call to arms event
  • For The Glory
  • Hearts of Iron III
  • Heir to the Throne
  • Europa Universalis III Complete
  • March of the Eagles
For all the complaints about the portraits, let's also remember what CK1's portraits looked like. If you're thinking the early CK3 portraits are cartoonish or something out of a mobile game, you haven't seen anything.

 
  • 11
  • 4Like
  • 4
Reactions:

crowdemon

Colonel
48 Badges
May 13, 2013
1.111
822
  • Crusader Kings II
  • Cities: Skylines - After Dark
  • Europa Universalis IV: Pre-order
  • Europa Universalis IV: Res Publica
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Art of War
  • Crusader Kings II: Sword of Islam
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: The Republic
  • Crusader Kings II: Rajas of India
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: Charlemagne
  • Crusader Kings III
  • Stellaris - Path to Destruction bundle
  • Europa Universalis IV: Mandate of Heaven
  • Stellaris: Humanoids Species Pack
  • Stellaris: Apocalypse
  • Europa Universalis IV: Rule Britannia
  • Crusader Kings II: Monks and Mystics
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Stellaris: Distant Stars
  • Stellaris: Megacorp
  • Crusader Kings II: Holy Fury
  • Prison Architect
  • Stellaris: Ancient Relics
  • Stellaris: Digital Anniversary Edition
  • Europa Universalis IV: Rights of Man
  • Crusader Kings II: Reapers Due
  • Stellaris: Leviathans Story Pack
  • Cities: Skylines
  • Europa Universalis IV: El Dorado
  • Crusader Kings II: Way of Life
  • Europa Universalis IV: Common Sense
  • Crusader Kings II: Horse Lords
  • Crusader Kings II: Conclave
  • Europa Universalis IV: Call to arms event
  • Europa Universalis IV: Cossacks
  • Cities: Skylines - Snowfall
  • Europa Universalis IV: Mare Nostrum
  • Stellaris
  • Crusader Kings II: Jade Dragon
  • Stellaris: Synthetic Dawn
  • Europa Universalis IV: Third Rome
Informative dairy, thanks for sharing one part of development.
 
  • 1Like
Reactions: