Open/Closed betas before release

[Spidee]

Hello,

I know one of the PDX folks have commented on a steam, that while open betas are great for finding bugs and broken things, they are a nightmare to manage, because 90% of the things, that get reported are duplicates, or the quality of the report is low, lackluster, doesn't even state steps to duplicate.

What is your stance on closed betas, where you would invite 10-20 people, your hardcore players, who would undergo a training in basic QA and how to fill in the reports, they could cover all of your single player testing. Obviously NDAs would be signed and I am pretty sure, that there are dedicated fans, who would jump at the opportunity for no costs to help make this great game.

Have you considered this in the recent time? From the top of my head I can think of the guy, who is making deep dives into performance, the person, who made a huge deep dive into machine empires economy, I am pretty sure some of the modders would also be a great asset to purely QA, not to interfere with your vision.

Thoughts or responses?
Cheers

 

[Pyoro]

I'm kinda dubious the issue is finding bugs (which betas are good for). It seems to me it's more of an issue of time to fix them. So adding a beta would perhaps only divert more resources from actual QA being done.

But obviously "from outside" that's all fairly impossible to tell so I can't say I have much of an opinion either way ^^;

 

[Juboboman]

I honestly don't get it. Because it seems like within 1 week of release we have a hotfix up, by week 2 we have a rolling beta server up and from then on we have continual updates that are substantial. So whatever process they have for identifying and fixing bugs post release seems to work quick enough so that they can pump out fixes with at most a 2 week lag. So why would not that not work for either open or closed beta testing for major releases? At the very least for the month before release. So that way all the fixes that would normally be rolled up in the next week or 2 (when I expect the game will finally be at a largely playable state, minus AI issues) can be ready for release itself?

 

[sortulv]

Must as it saddens me to say this, I think they already have a open beta - it just isn't noted down as such. It would probably have been more healthy for the community if the paid dlc didn't release alongside the functional patch changes, but rather came a little later once the worst bugs have been fixed...

 

[Duuk]

In my experience with closed betas, 90% of people invited to beta (including, sadly, myself for HOI ) wind up missing things or just not having the time to see things or test things. Volunteers can't possibly have time to test things extensively, even well meaning ones. Open betas always work better because more eyes = more possible chances to see things.

 

[Blurb]

What is your stance on closed betas, where you would invite 10-20 people, your hardcore players, who would undergo a training in basic QA and how to fill in the reports, they could cover all of your single player testing. Obviously NDAs would be signed and I am pretty sure, that there are dedicated fans, who would jump at the opportunity for no costs to help make this great game.

If I recall correctly, a few streamers had received some sort of early access for MegaCorp a week or two prior to official release.
Doesn't sound too far from what you're suggesting, but I'd not expect a lot of people to put out quality work (proper QA is *work*) for free.

 

[GuildenSpur]

I have suggested this at the start of Stellaris. Now, I work in IT and develop and have managed as well.
When I saw all the changes they did for this version, I really felt like there where going to be some issues.
There are a lot of possibilities for empires in the code so it's hard to have them all covered.
That's where the extensive playtesting comes in and you need most combinations to get a feel of how good your code is.
My suggestion:
- For every major version, create a beta branch
- Have beta testers play and report bugs. Only use beta tester who play a lot and have the time to report the bugs.\
They should also avoid reporting known bugs. Saves dev time.
- After a branch is getting stable, the patch/version can be released\
- Use branch for more hotfixing
- Close branch when a stable enough release is reached.
- If a DLC is to be released, same thing alongside the this beta test.

People wanting to play bleeding edge get to help PDX.
People who *hate* any type ofobugs can wait until the fixes are done.
PDX get's free testers. Testers get early access. Customers get a stable release. Dev's get the credit they deserver. Project lead doesn't need to come over to apologize.
What's the difficulty?
As I said, I suggested this before. This method or something similar benefits all and fixes to a large extent what is wrong with the current model.

 

[Custodin]

I think 2.2 would have been received much better if it were released as early-access. Give this early-access pass to people who pre-order the expansion, I bet you'd also get more pre-orders that way (I personally never pre-order unless there is a good reason to do so). It should come with a disclaimer "non final version, expect bugs, glitches, crashes.." to make it clear what you are possibly signing up for. If you don't want banana software, then don't pre-order.
I'm sure there's quite a number of people who'd rather have the unfinished product (or beta version) now and still have the more stable version 1 month later, as compared to have no update now and the same version they would otherwise have got 1 month later.. Bugs, performance issues, AI stupidity etc. would all be seen more favorably in a pre-release version where you pretty much have to expect stuff like this, as long as they are fixed for official release.
I would also argue that the people in that group should be on average more enthusiastic about filing useful bug reports than your average user (the difference might not be so big though).

Then there are those (probably most people) that want a stable, working product and are not interested in beta-testing buggy versions with game-breaking bugs.
With having the early-access phase running prior to the final release, at the very least you'd have a much better overview on existing issues. And given the fact that we are usually getting substantial improvements / bugfixes within a few weeks after releasing a big update, you'd get the release version probably to a state that most people would be content (if not happy) with.

I count myself to the first group (I could not wait for 2.2 to be released, and actually had lots of fun with it even if it has lots of issues), but also fully understand people in group 2 that want to buy products once they are ready and not before.
Wouldn't the winning move be to give both groups what they want? Or maybe I am missing something obvious that would make this approach not possible..

 

[Duuk]

The howling would be exactly the same. Exactly. the. same.

 

[Molikroth]

The howling would be exactly the same. Exactly. the. same.

I really wish I could sit here and disagree with you. I really really do.

 

[BlackUmbrellas]

A lot of the issues I've seen reported are ones either:

1. I haven't experiences myself, meaning they're not universal (and thus lower priority or might've slipped past Paradox)
2. I have experienced, but have worked around just fine even without resorting to mods and cannot understand being called "game-breaking".

 

[AlphaAsh]

This has been asked for several times since the release of the game. The answer has been no. Reasons. I didn't agree with them then, and I still don't now.

I've been involved in closed and open betas with a number of game studios. It was always to their benefit, although closed betas tended to be more productive than open ones.

 

[Custodin]

To expand a bit why I brought up this early-access point:

Paradox strategy as far as I know is to always release major updates together with (major) DLC. This way they'll sell more DLC units than they would otherwise have if the DLC was released in isolation (this is just an educated guess, I don't have facts or numbers to back this up).

It was my understanding that this is one of the main reasons why there is no open Beta. I think it makes sense from a business perspective, they don't want people to play the Beta without the DLC and then possibly deciding after playing without said DLC, that they don't need it all that much (historically, most updates + improvements go to the free update rather than the DLC). And maybe they'll get it later when it's on sale, if at all.. This would obviously be bad for business.

The other big reason against a open Beta is that reviewing the bug reports of a large group of testers can get quite inefficient. You don't necessarily want as many beta testers as possible, because you don't have unlimited amounts of time to review bug reports, identify duplicate reports vs. new/unique reports etc. It's true that you are more likely to find certain kind of bugs (like hardware related stuff) but more is not always better - you still need to find + interpret that bug report amongst hundreds reports referring to issues already fixed in the internal build they are running.

So I'm all for the idea with the Beta, but it's much more likely to happen when it doesn't conflict with PDX business strategy. Calling it "early access" could address both points I think.

 

[Athmet]

I think 2.2 would have been received much better if it were released as early-access. Give this early-access pass to people who pre-order the expansion, I bet you'd also get more pre-orders that way (I personally never pre-order unless there is a good reason to do so). It should come with a disclaimer "non final version, expect bugs, glitches, crashes.." to make it clear what you are possibly signing up for. If you don't want banana software, then don't pre-order.
I'm sure there's quite a number of people who'd rather have the unfinished product (or beta version) now and still have the more stable version 1 month later, as compared to have no update now and the same version they would otherwise have got 1 month later.. Bugs, performance issues, AI stupidity etc. would all be seen more favorably in a pre-release version where you pretty much have to expect stuff like this, as long as they are fixed for official release.
I would also argue that the people in that group should be on average more enthusiastic about filing useful bug reports than your average user (the difference might not be so big though).

Then there are those (probably most people) that want a stable, working product and are not interested in beta-testing buggy versions with game-breaking bugs.
With having the early-access phase running prior to the final release, at the very least you'd have a much better overview on existing issues. And given the fact that we are usually getting substantial improvements / bugfixes within a few weeks after releasing a big update, you'd get the release version probably to a state that most people would be content (if not happy) with.

I count myself to the first group (I could not wait for 2.2 to be released, and actually had lots of fun with it even if it has lots of issues), but also fully understand people in group 2 that want to buy products once they are ready and not before.
Wouldn't the winning move be to give both groups what they want? Or maybe I am missing something obvious that would make this approach not possible..

I like the idea: would help getting good pre-order from people willing to go through this and then once ready, the rest will get the DLC with quality they are looking for straight out of the box sort of speak.

 

[Tacticus101]

A delay might have been sensible, given some of the 2.2 changes felt a little rushed. But i dont think a beta would have changed much. Open Betas are very unreliable, and would have taken ages to solve the issues, resulting in a game that has not even been released by now. A Closed Beta is used for some games, and hasnt seemed to help a great deal, plus has the same issues with delay.

Really simply, i would prefer to play a bugged but playable game in December and give feedback to let the Devs improve it over the course of the next few months than delay the release by 2 months to fix some, but not all, of the issues.

Paradox strategy as far as I know is to always release major updates together with (major) DLC. This way they'll sell more DLC units than they would otherwise have if the DLC was released in isolation (this is just an educated guess, I don't have facts or numbers to back this up).

It was my understanding that this is one of the main reasons why there is no open Beta. I think it makes sense from a business perspective, they don't want people to play the Beta without the DLC and then possibly deciding after playing without said DLC, that they don't need it all that much (historically, most updates + improvements go to the free update rather than the DLC). And maybe they'll get it later when it's on sale, if at all.. This would obviously be bad for business.

Paradox release DLC with patches because it makes it easier to divide things up into additions to the game (DLC) and changes to existing features (Patch). Releasing a DLC with a major patch is practical, you can add new features that make use of any changed or expanded mechanics, and you will always have a patch with a DLC because you need to ensure every event and mechanic works with any DLC features.

Paradox have made it clear that the game is supposed to be entirely playable without the DLC, you dont need it.

 

[Secret Master]

The howling would be exactly the same. Exactly. the. same.

 

[DreadLindwyrm]

Adding a closed beta of around 20 people wouldn't necessarily do much - bear in mind that 20 extra people is tiny compared to the number of players when the game goes live.

Perhaps it'll add in a few extra setups, and result in a few more bugs being caught and passed up for resolution - but it doesn't mean there will be, and it doesn't mean that even if the beta testers find them that they'll be replicated in house and thus be able to be resolved.

There have been (and may still be) beta testers selected from the community for various games, and not all of them always turn in bug reports that are usable. Or any.
Every beta tester is a possible leak as well, so they have to be careful with the vetting.

And then there are things that might be considered a bug/unacceptable by the wider community, but that the beta testers either don't discover or don't consider a significant bug.
Then what we get (in addition to "internal QA are useless") is "the beta testers are suckups and yesmen and should be replaced!".

 

[Rithral]

The howling would be exactly the same. Exactly. the. same.

If it was an opt in patch, I don’t think it would be the same.
2.2.0 would be the beta optional opt in released in early December.
2.2.0.1 would be the first hot-fix for the opt in beta
2.2.0.2 would be the second hotfix that dropped before winter break for the opt in beta.
2.2.1 would have been the full release and would have dropped today with the MegaCorp specific Paid content.

 

[Duuk]

If it was an opt in patch, I don’t think it would be the same.
2.2.0 would be the beta optional opt in released in early December.
2.2.0.1 would be the first hot-fix for the opt in beta
2.2.0.2 would be the second hotfix that dropped before winter break for the opt in beta.
2.2.1 would have been the full release and would have dropped today with the MegaCorp specific Paid content.

You must be new on the internet if you think people wouldn't howl at the moon about OMG BROKEN AND THEN YOU WENT ON VACATION just because it was an optional early-access thing with a warning label.

 

[Ryika]

I honestly don't get it. Because it seems like within 1 week of release we have a hotfix up, by week 2 we have a rolling beta server up and from then on we have continual updates that are substantial. So whatever process they have for identifying and fixing bugs post release seems to work quick enough so that they can pump out fixes with at most a 2 week lag. So why would not that not work for either open or closed beta testing for major releases? At the very least for the month before release. So that way all the fixes that would normally be rolled up in the next week or 2 (when I expect the game will finally be at a largely playable state, minus AI issues) can be ready for release itself?

Many companies have the equivalent of a "feature lock" when it comes to fixing things shortly before release. You simply don't do it, because unexpected consequences of those fixes could lead to problems that are much bigger than the bugs that you're trying to fix. So instead, you release the build that has the bugs but is is confirmed to work overall and move on to an internal build where you continue to fix things and test the implications that those fixes have for the rest of the game.

I don't know how Paradox does things, but it seems like it could be an explanation for the quick "post-launch hotfix", stuff was in the work internally and was pushed after it has been verified as mostly stable.