Open/Closed betas before release

  • We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

Beardy-Bard

Private
Dec 25, 2018
14
0
I know one of the PDX folks have commented on a steam, that while open betas are great for finding bugs and broken things, they are a nightmare to manage, because 90% of the things, that get reported are duplicates, or the quality of the report is low, lackluster, doesn't even state steps to duplicate.

I dont understand why duplicate reports are a bad thing?
I see it like voting
lets say you have 100 Beta testers
80 report a AI Bug
30 report a Resolution Bug
5 Report a performance problem
and so on

wont that help prioritize?
 

Spidee

Major
74 Badges
Jan 1, 2013
686
433
  • Crusader Kings II
  • Europa Universalis IV: Rights of Man
  • Europa Universalis IV: Common Sense
  • Crusader Kings II: Horse Lords
  • Europa Universalis IV: Cossacks
  • Crusader Kings II: Conclave
  • Stellaris: Humanoids Species Pack
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Ancient Relics
  • Stellaris Sign-up
  • Hearts of Iron IV: Cadet
  • Hearts of Iron IV: Expansion Pass
  • Crusader Kings II: Way of Life
  • Tyranny: Archon Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Crusader Kings II: Monks and Mystics
  • Stellaris: Lithoids
  • Europa Universalis IV: Mandate of Heaven
  • BATTLETECH
  • Surviving Mars
  • Hearts of Iron IV: Death or Dishonor
  • Stellaris: Synthetic Dawn
  • Europa Universalis IV: Cradle of Civilization
  • Age of Wonders III
  • Hearts of Iron IV: La Resistance
  • Crusader Kings III
  • Age of Wonders: Planetfall Deluxe edition
  • Shadowrun: Dragonfall
  • Stellaris: Nemesis
  • Europa Universalis 4: Emperor
  • Stellaris: Distant Stars
  • Age of Wonders: Planetfall
  • Europa Universalis IV: Rule Britannia
  • Europa Universalis IV
  • Stellaris: Federations
  • Stellaris: Necroids
  • Pillars of Eternity
  • Crusader Kings II: Holy Fury
  • Shadowrun: Hong Kong
  • Surviving Mars: Digital Deluxe Edition
  • Stellaris: Apocalypse
  • Shadowrun Returns
  • Victoria 2
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Warlock: Master of the Arcane
  • 500k Club
  • Europa Universalis IV: El Dorado
I dont understand why duplicate reports are a bad thing?
I see it like voting
lets say you have 100 Beta testers
80 report a AI Bug
30 report a Resolution Bug
5 Report a performance problem
and so on

wont that help prioritize?
Because you have a database of 8000 reports, which covers 687 bugs/issues. 12 reports on average, all worded differently, that means, that actually someone has to consolidate the 8000 into the 687.
because there would be:
AI not attacking, when it should.
During war, 2/3 of AI fleets are defending a planet.
While attacking the AI, 1 fleet, representing 40% of their power was on pirate duty.
AI declared war on superior strength me they did not attack.
On W10, graphics mods enabled, after declaring war on 2 AI empires, they just stood still. I think they try to follow each other.

Not to mention, spelling mistakes, or things on displaying properly or calculated properly would be there dozens of times. Wouldn't really help with priority, because, except the sliders not aligning, those things don't make the game unplayable.
 

Onkel_Bums

Second Lieutenant
4 Badges
Jul 8, 2017
122
1
  • Stellaris
  • Stellaris: Apocalypse
  • Stellaris: Distant Stars Pre-Order
  • Stellaris: Megacorp
In my personal opinion it would be beneficial to get the community to test updates, that are this game changing, before the release.

If such an update is accompanied by a DLC, maybe just have the pre-order customers be included in the beta. Pre orders would spike, while people are informed that it's likely buggy.
However, only the "free patch" should be subject of that beta, not the DLC. Core changes could get more feedback, before the DLC is released. PDX would still make (maybe less, but still) money, but it would be perceived as a more honest approach.

This should result in a higher quality free patch as a better foundation for the accompanying DLC, which will likely also sell more because the better quality is reflected in reviews and recommendations.
If DLC content is still buggy by that time, at least the people who did not buy it will still give good reviews because the free patch works.

This goes in line with the following:

Must as it saddens me to say this, I think they already have a open beta - it just isn't noted down as such. It would probably have been more healthy for the community if the paid dlc didn't release alongside the functional patch changes, but rather came a little later once the worst bugs have been fixed...
 
Last edited:

Vitruvian Guar

Major
11 Badges
Apr 12, 2017
601
1
  • Stellaris: Synthetic Dawn
  • Cities: Skylines
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Stellaris - Path to Destruction bundle
  • Stellaris: Humanoids Species Pack
  • Stellaris: Apocalypse
  • Stellaris: Distant Stars
Well, while the idea is definitely good I predict, that it's not going to happen. For the same reason other good ideas like world peace or getting rid of corruption are not happening.

Currently the community is dissapointed after people bought the DLC. With the beta test dissapointed people simply won't buy the payed content. In the first scenario company gets more money and worse attitude from the community. In the second, better relationship with the customers but less money. And no matter how much we want to believe that companies are interested in the customers interests, any time they conflict with the profits - customers interests will be sacrificed. Not because the company is evil, just because market economy optimize for earning money rather than anything else.
 

Onkel_Bums

Second Lieutenant
4 Badges
Jul 8, 2017
122
1
  • Stellaris
  • Stellaris: Apocalypse
  • Stellaris: Distant Stars Pre-Order
  • Stellaris: Megacorp
With the beta test dissapointed people simply won't buy the payed content.

There is a high probability that these are the same people who would also not buy the paid content at full price on release anyway, which I think weakens your argument a bit. And you miss that a better relationship with the community is not exclusive with less money. I'd argue just the opposite. Happy customers come back, and recommend.
 

Spidee

Major
74 Badges
Jan 1, 2013
686
433
  • Crusader Kings II
  • Europa Universalis IV: Rights of Man
  • Europa Universalis IV: Common Sense
  • Crusader Kings II: Horse Lords
  • Europa Universalis IV: Cossacks
  • Crusader Kings II: Conclave
  • Stellaris: Humanoids Species Pack
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Ancient Relics
  • Stellaris Sign-up
  • Hearts of Iron IV: Cadet
  • Hearts of Iron IV: Expansion Pass
  • Crusader Kings II: Way of Life
  • Tyranny: Archon Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Crusader Kings II: Monks and Mystics
  • Stellaris: Lithoids
  • Europa Universalis IV: Mandate of Heaven
  • BATTLETECH
  • Surviving Mars
  • Hearts of Iron IV: Death or Dishonor
  • Stellaris: Synthetic Dawn
  • Europa Universalis IV: Cradle of Civilization
  • Age of Wonders III
  • Hearts of Iron IV: La Resistance
  • Crusader Kings III
  • Age of Wonders: Planetfall Deluxe edition
  • Shadowrun: Dragonfall
  • Stellaris: Nemesis
  • Europa Universalis 4: Emperor
  • Stellaris: Distant Stars
  • Age of Wonders: Planetfall
  • Europa Universalis IV: Rule Britannia
  • Europa Universalis IV
  • Stellaris: Federations
  • Stellaris: Necroids
  • Pillars of Eternity
  • Crusader Kings II: Holy Fury
  • Shadowrun: Hong Kong
  • Surviving Mars: Digital Deluxe Edition
  • Stellaris: Apocalypse
  • Shadowrun Returns
  • Victoria 2
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Warlock: Master of the Arcane
  • 500k Club
  • Europa Universalis IV: El Dorado
Well, while the idea is definitely good I predict, that it's not going to happen. For the same reason other good ideas like world peace or getting rid of corruption are not happening.

Currently the community is dissapointed after people bought the DLC. With the beta test dissapointed people simply won't buy the payed content. In the first scenario company gets more money and worse attitude from the community. In the second, better relationship with the customers but less money. And no matter how much we want to believe that companies are interested in the customers interests, any time they conflict with the profits - customers interests will be sacrificed. Not because the company is evil, just because market economy optimize for earning money rather than anything else.
I partially disagree with this. My modus operandi is: you mess up, you go into a "time out". After SW7, I did not watch Rogue one, after Last Jedi, I will not watch SW9.
Ripples in the community lose you money in the long haul. You can't bet, that eventually we will forgive, forget and buy anything blindly.
 

Korruna

Private
30 Badges
Mar 8, 2018
11
1
  • Surviving Mars
  • Crusader Kings III
  • Stellaris: Federations
  • Stellaris: Lithoids
  • Stellaris: Ancient Relics
  • Prison Architect
  • Stellaris: Megacorp
  • Cities: Skylines Industries
  • Stellaris: Distant Stars
  • Cities: Skylines - Parklife
  • Stellaris: Apocalypse
  • Stellaris: Humanoids Species Pack
  • Cities: Skylines - Green Cities
  • Cities: Skylines - Mass Transit
  • Stellaris - Path to Destruction bundle
  • Cities: Skylines - Natural Disasters
  • Stellaris: Leviathans Story Pack
  • Stellaris: Digital Anniversary Edition
  • Tyranny: Archon Edition
  • Cities: Skylines - Snowfall
  • Cities: Skylines - After Dark
  • Pillars of Eternity
  • Cities: Skylines
  • Magicka
  • Crusader Kings II
  • Stellaris: Synthetic Dawn
  • Age of Wonders III
  • Stellaris
  • Magicka 2
  • Europa Universalis IV
I know one of the PDX folks have commented on a steam, that while open betas are great for finding bugs and broken things, they are a nightmare to manage, because 90% of the things, that get reported are duplicates, or the quality of the report is low, lackluster, doesn't even state steps to duplicate.

If what PDX said was true, then I don't know how they manage to comb through their live bug reports. There is a greater order of magnitude more bug reports on a live game than a beta one. With regards to reporting bugs, PDX puts many barriers (inconveniences) in place which makes it a hassle. There's no in-game bug reporting button, so you have to tab out and write up a report on their website. (Which you have to log into) Furthermore, it requires information like version#, checksum, and in some cases the player needs to find where their screenshots and saved games are kept.

These don't seem like much of an inconvenience, but the chances are that a majority of the people who might report a bug will give up upon immediately learning there is no bug reporting button and that they have to go to a website to do it. Having a beta also culls out a large amount of people, especially if you have to opt into it.

What you're describing is basically a Friends & Family alpha, which PDX probably already has. I have participated in a few of those and one of the biggest problems that you run into is leaks and security. Close friends and families of developers can be trusted, but can you trust some "high profile" member of the community to not share what's happening to his friends secretly on his clan's Discord? I've seen many futile measures take place to tighten up leaks and at the end of the day it's almost impossible to keep strangers from sharing secrets with their friends.

A bigger problem (imo) is the fact that it's really exploitative, taking advantage of the passion of gamers. Paradoxically (ha!), the more professional and skilled a group of volunteers gets at reporting and diagnosing bugs, the more ethically questionable it becomes. A secret group of volunteer testers is a thankless, perkless job that will often be demoralized by the bugs that inevitably slip through which the community will lambast about.

In my opinion, the fairest way to do a beta is just to have it be NDA free and open. (Or at the worst, opt-in) Good bug reports will rise to the top, and if the program goes on long enough it becomes easier over time for developers to recognize members of the community that are proficient at reporting things.
 
Last edited:

Vitruvian Guar

Major
11 Badges
Apr 12, 2017
601
1
  • Stellaris: Synthetic Dawn
  • Cities: Skylines
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Stellaris - Path to Destruction bundle
  • Stellaris: Humanoids Species Pack
  • Stellaris: Apocalypse
  • Stellaris: Distant Stars
There is a high probability that these are the same people who would also not buy the paid content at full price on release anyway, which I think weakens your argument a bit.

What your reasoning for this point? Anyway compare this situations in your mind:

You've been seeing teasers for a couple of mounths lost the ability to play current version of the game and crave for new one with all this juicy payed content. And then:

1) You get free access for beta test and find out that the game is pretty bugged and the payed content isn't that impressive.
2) You meet the release date without any insight of the game state.

I would say that first scenario will leave you a bit dissapointed, and the second rather hyped. And people tend to spend money when they are hyped.

And you miss that a better relationship with the community is not exclusive with less money. I'd argue just the opposite. Happy customers come back, and recommend.

I'm not claiming that the profit of a company is not usually positively correlated with the community attitude. It would be obviously false. Of course relationship with the customers and reputation matters. It's the reason why such good company as PDX exists in the first place. I'm claiming that sometimes there is a conflict between customer interests and company profits and in those occasions customers satisfaction will be always sacrificed in favour of profits. And that the situation with beta test of paid features is exactly this one. Otherwise how would you explsin our situation?

I partially disagree with this. My modus operandi is: you mess up, you go into a "time out". After SW7, I did not watch Rogue one, after Last Jedi, I will not watch SW9.
Ripples in the community lose you money in the long haul. You can't bet, that eventually we will forgive, forget and buy anything blindly.

Your strategy is a valid one, yet it doesn't really matter. What matters is the median strategy of all customers and it is turning out to be: getting hyped, buying dlc, getting dissapointed, feeling grateful for the patches and then getting hyped again.
Utopia, Apocalypse, now Megacorp... Seems that you indeed can bet that the customers will forget. Also, too bad you missed Rogue One, it's much better than Force Awakens.
 

Spidee

Major
74 Badges
Jan 1, 2013
686
433
  • Crusader Kings II
  • Europa Universalis IV: Rights of Man
  • Europa Universalis IV: Common Sense
  • Crusader Kings II: Horse Lords
  • Europa Universalis IV: Cossacks
  • Crusader Kings II: Conclave
  • Stellaris: Humanoids Species Pack
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Ancient Relics
  • Stellaris Sign-up
  • Hearts of Iron IV: Cadet
  • Hearts of Iron IV: Expansion Pass
  • Crusader Kings II: Way of Life
  • Tyranny: Archon Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Crusader Kings II: Monks and Mystics
  • Stellaris: Lithoids
  • Europa Universalis IV: Mandate of Heaven
  • BATTLETECH
  • Surviving Mars
  • Hearts of Iron IV: Death or Dishonor
  • Stellaris: Synthetic Dawn
  • Europa Universalis IV: Cradle of Civilization
  • Age of Wonders III
  • Hearts of Iron IV: La Resistance
  • Crusader Kings III
  • Age of Wonders: Planetfall Deluxe edition
  • Shadowrun: Dragonfall
  • Stellaris: Nemesis
  • Europa Universalis 4: Emperor
  • Stellaris: Distant Stars
  • Age of Wonders: Planetfall
  • Europa Universalis IV: Rule Britannia
  • Europa Universalis IV
  • Stellaris: Federations
  • Stellaris: Necroids
  • Pillars of Eternity
  • Crusader Kings II: Holy Fury
  • Shadowrun: Hong Kong
  • Surviving Mars: Digital Deluxe Edition
  • Stellaris: Apocalypse
  • Shadowrun Returns
  • Victoria 2
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Warlock: Master of the Arcane
  • 500k Club
  • Europa Universalis IV: El Dorado
Your strategy is a valid one, yet it doesn't really matter. What matters is the median strategy of all customers and it is turning out to be: getting hyped, buying dlc, getting dissapointed, feeling grateful for the patches and then getting hyped again.
Utopia, Apocalypse, now Megacorp... Seems that you indeed can bet that the customers will forget. Also, too bad you missed Rogue One, it's much better than Force Awakens.
Well......let me re-phrase. Disney did not get any revenue from me for Rogue One....
 

Rithral

Major
35 Badges
Dec 16, 2002
757
160
Visit site
  • Stellaris: Humanoids Species Pack
  • Hearts of Iron IV: Field Marshal
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Stellaris - Path to Destruction bundle
  • BATTLETECH
  • Hearts of Iron IV: Death or Dishonor
  • Stellaris: Synthetic Dawn
  • Hearts of Iron IV: Expansion Pass
  • Hearts of Iron IV: Colonel
  • Stellaris: Apocalypse
  • BATTLETECH - Digital Deluxe Edition
  • Stellaris: Distant Stars
  • BATTLETECH: Flashpoint
  • Stellaris: Megacorp
  • Hearts of Iron IV: Expansion Pass
  • BATTLETECH: Season pass
  • BATTLETECH: Heavy Metal
  • Semper Fi
  • Crusader Kings II
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Sword of Islam
  • For the Motherland
  • Hearts of Iron III
  • Victoria: Revolutions
  • Hearts of Iron II: Armageddon
  • Victoria 2
  • 500k Club
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Hearts of Iron IV: Cadet
You must be new on the internet if you think people wouldn't howl at the moon about OMG BROKEN AND THEN YOU WENT ON VACATION just because it was an optional early-access thing with a warning label.

The number of people with the rolling beta 2.2.3 is a fraction of those running the mainline 2.2.2 patch.
Only a small fraction of people would elect to update to an optional beta. Those who found the beta “broken” would have the option to swapping back.
 

Duuk

Reformed Badboy
23 Badges
Oct 16, 2001
6.137
1.402
  • Majesty 2
  • Europa Universalis III: Collection
  • Crusader Kings II: Holy Knight (pre-order)
  • Cities: Skylines
  • 500k Club
  • Victoria 2: Heart of Darkness
  • Victoria 2: A House Divided
  • Victoria 2
  • Rome Gold
  • Victoria: Revolutions
  • Europa Universalis III Complete
  • Majesty 2 Collection
  • Hearts of Iron Anthology
  • Europa Universalis III Complete
  • Heir to the Throne
  • Europa Universalis III Complete
  • Europa Universalis III
  • Deus Vult
  • Crusader Kings II: Sword of Islam
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: The Republic
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II
The number of people with the rolling beta 2.2.3 is a fraction of those running the mainline 2.2.2 patch.
Only a small fraction of people would elect to update to an optional beta. Those who found the beta “broken” would have the option to swapping back.
But if it was the ONLY way to get the new content (ie: a pre-release of a new expansion, even if it didn't allow people to play the DLC-only content) the opt-in would be much, much more universal.

And the howling would be legendary.