Paradox, include CB testing into your QA

[Alblaka]

To be fair the amount of new features and redone old features was staggering in this patch and the patches 1.1-1.4 were working quite good on release.

Hmmm, this is a fair argument I didn't consider yet.
I mean, if you intend to release a contant width far exceeding your usual capacities, you should take measures to ensure Quality is not replaced by Quantity (i.e. temporarily increasing QA budget or allowing a longer test phase), but I wouldn't be too disappointed hearing PDX couldn't afford either. Means they probably should have reduced the scope of the patch though (after all, I would really have preferred to either have a solidly thought-out Unity system, OR nice and diverse hiveminds, over having both semi-functional).

Bringing in earlier patches though... maybe my critique is indeed too focussed on 1.5 and does not respect the fact QA has been keeping up in previous patches.
Which begs the question whether future patches will be '1.4' or '1.5' size. As, in former case, QA might be fine continuing, but in latter, ambitious, case, the core of my open letter still needs to be adressed.

Also I cannot forgive breaking steam achievements.

I agree that this is highly annoying for some, but I would be fair and point out it's hard to test steam achievements in a closed developement area not linked to Steam.
Albeit then again Achievements shouldn't have been broken in first place, thus this is an actual case of 'upredictable bug' that belongs into a hotfix, but is 'ethically acceptable' to have on release day.

You accuse them of being incompetent and having the desire to do QA.

How can you even missread a logical conclusion chain THAT hard?

• Stellaris QA is not properly playtesting the full width of features added in a new patch.

• This implies the QA team is either incompetent, or understaffed.

• In the benevolent/realistic assumption that the QA team is actually giving it's best effort and is competent, this implicates Paradox is not able/willing to hire enough QA staff members to properly provide test coverage for a game of this feature width and depth.

I CLEARLY state that the playtesting wasn't enough for the width of features, because EVIDENTLY it wasn't. I am NOT accusing them of being incompetent, only of the result of their work being unfit for the scope of the content.
Furthermore I then deduce what likely causes this 'lack of QA' could have, with the two obvious answers being that EITHER OR.
And then I rather clearly state that I do not think it's 'incompetency', but instead a lack of invested ressources.

It's an detailed explanation of my train of thoughts that lead to the initial statement

In the assumption that Paradox doesn't have the intention/budget to fund a proper QA

Honestly, it takes A LOT of missinterpretation to turn that into 'You are accusing them of being incompetent'.

I've seen it mentioned a number of times on PDX forums, and others, that a number - not necessarily all - of bugs are found before release but cannot be fixed for a number of reasons. You pointed out yourself one issue that has seeminly been fixed in the beta patch. Perhaps the priority was on other issues; ones which prevent the game from actually being played. I can't say as I don't know. But you can't know for certain which bugs PDX were or weren't aware of prior to release.

Yes, this is a possibility. However, it's another issue for itself:

• Did the devs know of the bugs, but didn't find time to fix them?
• Why didn't they find the time? Was there some critical event shortly before released that chomped up manpower (i.e. Truck-factor)? Why wouldn't they delay the release for that, or at least point out to the public that such a critical event happened and may have affected the game, if they cannot delay.
• Did they not find the time because they priorized adding new untested content over assuring quality? This would be, to me, a fatal flaw in decisionmaking for any IT company (albeit sadly not a rare occurence).
• Or is the issue that the bug was found TOO LATE to fix it (which is very likely, giving the 1.5.1 timing)?
• In which case it's again a QA issue of not having enouhg time to find all/most of these kind of bugs before the devs are out of time to fix them. Codefreeze is a thing, and it's a confirmed-to-be-employed strategy in PDX.

I've read evidence with respect to other titles - EU for instance - that there might be a small number of players who have been chosen to do some closed beta testing of patches.

I would be glad to read any of that evidence. If it's more than just your casual forum rumor, it would be a good angle to come from.
Albeit I would beg to differ between 'CB with a small amount of players' and 'unpaid QA'. The core of CB is the fact that it is not public, but contains 'large' numbers of actual players.
Putting 5 youtubers on CB duty is unpaid QA, having 100 players playtest is CB.
(At least in my understanding and definition of the term, albeit naturally any company can do and label their CB's however they want)

 

[General Retreat]

If I had to guess why 'obvious' bugs slip through, it's probably because you've got 6 or so people on the team at the moment (from what I remember of a recently issued posted video of a talk by the team programmer) with a presumably smaller dedicated QA unit.

Because of divergent gameplay options, most of the late game issues don't become apparent until 10s of hours into a given run. There isn't really an easy way to cut that manual workload down by unit testing either, because sneaky console tricks to fast forward to certain points can compromise the testing environments by overlooking bugs that may occur incrementally over the course of a run. The majority of bugs are also highly context sensitive and need a specific set of steps to replicate.

That's why bugs seem obvious to players after 10 hours but they haven't necessarily been caught by QA. As a player you're going to have preferences about how to play, and made a beeline for a specific configuration that excites you. Thousands of others will have done the same in varying permutations. All those bugs will inevitably be found, but the public is dumping a combined total of 100,000 play hours over the space of a week, when a single QA tester can reasonably manage maybe 40-50.

It's always good to expect more, but frankly it's demoralizing for all parties involved to expect miracles. If 100% prevention isn't possible given the nature of the beast, I'm happy to settle for responsive bug fixing and hotpatching.

 

[Alblaka]

That's why bugs seem obvious to players after 10 hours but they haven't necessarily been caught by QA. As a player you're going to have preferences about how to play, and made a beeline for a specific configuration that excites you. Thousands of others will have done the same in varying permutations. All those bugs will inevitably be found, but the public is dumping a combined total of 100,000 play hours over the space of a week, when a single QA tester can reasonably manage maybe 40-50.

Indeed.
And that's why establishing a closed beta system actually worth it's name (aka, with 2-digit player counts), would remedy the issue.

 

[Derek Pullem]

Having been a closed beta back in the days of HoI2, Victoria CK and EU2 I can assure you that Closed Betas do not remove all bugs.

I suspect the problems (if there are any problems) with 1.5 are that its a MASSIVE update. It almost feels like a new game and frankly I'm liking it loads. The betas (if any are still there) are probably much reduced in number though as the CB group tends to have a 50% no show at appointment and probably 50-75% of the rest will be gone by release.

 

[Alblaka]

Having been a closed beta back in the days of HoI2, Victoria CK and EU2 I can assure you that Closed Betas do not remove all bugs.

Of course, nothing ever removes all bugs. That's basically a core assumption of anything that is IT.
But I still feel like a proper QA/CB would find all obvious and glaring bugs that replicate themselves in rather default use cases.

I suspect the problems with 1.5 are that its a MASSIVE update.

Might be a good reason.
However, if it's the truth, this would imply the dev team overstepped their 'possible to achieve whilst maintaining quality' scope in the pursuit of hype-able content. And, consequently, should ponder more harshly on any content selections for future patches (ot alternatively extend the QA to match the content scope).

 

[MrSoulbinder]

I absolutely agree with the OP. And I want to add that in my opinion Paradox favors the development of new stuff too much above fixing, balancing and fine tuning the current content. Everybody loves new content and in the end this is where new money comes from, I totally get that, but some form of balance must be reached.

The bugs that made it to the live version in this and previous patches are a strong indicator for insufficient QA. To some degree this is mitigated by the active mod community (I remember there were mods created to fix blatantly obvious issues in previous versions), but frankly this is not the job of mods or the community, but due diligence of the developer.

 

[Mendeth]

How can you even missread a logical conclusion chain THAT hard?

I CLEARLY state that the playtesting wasn't enough for the width of features, because EVIDENTLY it wasn't. I am NOT accusing them of being incompetent, only of the result of their work being unfit for the scope of the content.
Furthermore I then deduce what likely causes this 'lack of QA' could have, with the two obvious answers being that EITHER OR.
And then I rather clearly state that I do not think it's 'incompetency', but instead a lack of invested ressources.

It's an detailed explanation of my train of thoughts that lead to the initial statement


Honestly, it takes A LOT of missinterpretation to turn that into 'You are accusing them of being incompetent'.

Again we come back to the fundamental difference in our opinions; you see the existence of bugs as proof that the QA failed. I on the other hand would prefer not to leap to criticising the QA when we don't know what bugs were identified/reported. Frankly, I find it objectionable that you even brought up the word 'incompetency'. It's a tall - and in my opinion serious - claim when you don't have all the facts at hand. Your original premise is one for which you provide no evidence other than the existence of bugs, which proves nothing other than that those bugs exist.

Yes, this is a possibility. However, it's another issue for itself:

• Did the devs know of the bugs, but didn't find time to fix them?
• Why didn't they find the time? Was there some critical event shortly before released that chomped up manpower (i.e. Truck-factor)? Why wouldn't they delay the release for that, or at least point out to the public that such a critical event happened and may have affected the game, if they cannot delay.
• Did they not find the time because they priorized adding new untested content over assuring quality? This would be, to me, a fatal flaw in decisionmaking for any IT company (albeit sadly not a rare occurence).
• Or is the issue that the bug was found TOO LATE to fix it (which is very likely, giving the 1.5.1 timing)?
• In which case it's again a QA issue of not having enouhg time to find all/most of these kind of bugs before the devs are out of time to fix them. Codefreeze is a thing, and it's a confirmed-to-be-employed strategy in PDX.

Who knows? I don't have information pertaining to the dev and bug-reporting processes for 1.5, and I'd be willing to wager you don't either. Again though I'd say you're leaping to conclusions; just because some bugs weren't fixed doesn't mean they prioritised new content over quality. There could be any number of reasons why some issues were prioritised over others, and immediately assuming that the new content compromised QA is tenuous without evidence.

I would be glad to read any of that evidence. If it's more than just your casual forum rumor, it would be a good angle to come from.

Unfortunately I have neither the time nor the inclination to trawl through the forums. Also admittedly the posts in question are most likely months if not years old so wouldn't be of real value to current PDX/PDS practices. That said, whilst I'm confident I'm not misremembering and other titles have had closed beta testing on patches, it is possible that I'm incorrect.

 

[Vaar]

Having been a closed beta back in the days of HoI2, Victoria CK and EU2 I can assure you that Closed Betas do not remove all bugs.

I suspect the problems (if there are any problems) with 1.5 are that its a MASSIVE update. It almost feels like a new game and frankly I'm liking it loads. The betas (if any are still there) are probably much reduced in number though as the CB group tends to have a 50% no show at appointment and probably 50-75% of the rest will be gone by release.

Well, I don't think anyone is expecting them to remove all bugs, but the current release is unacceptable. Honestly, maybe closed beta is not the solution, but if it's not, the problem is that Paradox is knowingly releasing a game with major bugs. So, which is it? Does Paradox need more help finding bugs, or are they knowingly releasing buggy products?

I can tell you from experience I found at least 20 bugs in 10 hours. Among them, my precursor quest line broke, my Horizon Signal quest line broke, an Enigmatic Fortress item no longer can be used, and at the end of Synthetic Evolution, all Synthetics become enslaved with the species rights button unavailable pretty much ending my game.

I understand it's a big patch. The solution is not to release it early, take people's money and fix the mess later. The solution is to take more time and release it when you've fixed all major bugs. Some minor bugs are fine, but events not working and game ending bugs are not.

 

[Derek Pullem]

Of course, nothing ever removes all bugs. That's basically a core assumption of anything that is IT.
But I still feel like a proper QA/CB would find all obvious and glaring bugs that replicate themselves in rather default use cases.


Might be a good reason.
However, if it's the truth, this would imply the dev team overstepped their 'possible to achieve whilst maintaining quality' scope in the pursuit of hype-able content. And, consequently, should ponder more harshly on any content selections for future patches (ot alternatively extend the QA to match the content scope).

Honestly this gets thrown around after every Paradox release (and most other games I buy, but I buy alot of Paradox games).

What you get with Paradox is a commitment to continuous improvement - they don't walk away from their games. Like it or loathe it their track record is similar to the big OS providers. They occasionally screw up with a big patch / update release and have to scramble to correct it especially for major changes. Hell there are things in Microsoft Office which still drive me nuts 15 years after I first noticed them. I don't get me on to Android releases breaking perfectly good apps.

Stay with the program and you get a quality product - eventually. Just like Windows

 

[Slarkon]

What makes you think they don't have closed beta testing?
I know for a fact that other main titles do...

i know there where some youtbers ect who got early utopia. whatever.

but if there is a closed beta that is a bad thing

1. They ignored the feedback from the beta testers and launched anyway (not just in 1.5 but in all other previous updates)

2. The beta testers are not the teenagers who will play 16 / 17 hours a day and discover more bugs. 32 here btw and allot of my friends would simply no longer have the time to beta test for bugs as life gets in the way. If they are picking beta testers who are not putting in the hours then they might as well not do it as too many bugs are getting through.

 

[grommile]

Well, I don't think anyone is expecting them to remove all bugs, but the current release is unacceptable.

*looks at the early patch history of Stellaris*

If this release is unacceptable, how did you even stick around this long?

 

[Meathim]

The Paradox fanbase is dedicated enough and would be more than helpful.

Bug-hunting is one thing, but I have a feeling it would devolve into "why not X" fairly quickly. From my short time here I can safely say the fanbase is also divided and have wildly different ideas about what the game should be.

Quality QA has to be in-house, or with people they know so well it might as well be in-house. And if the company isn't structured to have closed betas it's probably a big job to implement it, not to mention the shift in thinking.

 

[DMFan]

Bug-hunting is one thing, but I have a feeling it would devolve into "why not X" fairly quickly.

I'm not taking sides here, anyway, in regards to your objection, you get the beta tester role to find bugs, not to ask for game changes.

 

[Alblaka]

you see the existence of bugs as proof that the QA failed.
[...]
Your original premise is one for which you provide no evidence other than the existence of bugs, which proves nothing other than that those bugs exist.

That is correct. However, as elaborated, these are bugs that reoccur with 100% reliability, and are directly present in the default use case of their respected feature.
Aka, they would occur during the very first integrative tests you run (in this case, an integrative test would be to plainly start a game of Stellaris and play it through until the feature's occurence).
Therefore, it's a reliable conclusion that QA actually testing the feature would, with a near-certain probability, have found the issue (or might have found the issue, as you implied earlier, but which should be considered irrelevant according to what I'll elaborate further below).

Of course, maybe near-certain is not 'proof enough' for you, but then I'm sorry and can't help you further.

Again though I'd say you're leaping to conclusions;

...
YOU pointed out a possibility.
I responded to that possibility by providing a list of possible explanations and how I would presumably react to them, to underline the fact that the possibility, if it were to exist, wouldn't exactly be something contrary to the original topic.

And now I'm the one jumping to 'conclusions'?
I'm elaborating possibilities according to what I currently know, I did not state 'Therefore X is Y' (which, to me, is the only statement worth the term 'conclusion').

Except for the part where I 'concluded', that your mentioned possibility of the bugs not being fixed despite being known, doesn't exactly make things anything better for the devs, nor us (for afromentioned reasons).

Bug-hunting is one thing, but I have a feeling it would devolve into "why not X" fairly quickly.

Yep, and that's a key criteria to remove beta-testers from their assigned priviledge.

What you describe is an issue of selecting the right people for the system, not the system itself.

 

[Arkona]

Bug-hunting is one thing, but I have a feeling it would devolve into "why not X" fairly quickly. From my short time here I can safely say the fanbase is also divided and have wildly different ideas about what the game should be.

Quality QA has to be in-house, or with people they know so well it might as well be in-house. And if the company isn't structured to have closed betas it's probably a big job to implement it, not to mention the shift in thinking.

No one needs excuses. They're not a charity, they're a business. If you want excuses I could give you as many as you want.

What PDX needs is an upgrade to the QA/bug-fixing system so that game crushing bugs don't show up at release day. It's obvious that the "Embrace faction unhappiness" bug, which comes up every time you embrace a faction, would have made itself present, if someone had spent more than three hours testing the game. So, the conclusion is that either someone should get fired for incompetency, or they're not doing a great job overall because of understaffing or problems of an organizational nature.

 

[fofferfoffer]

Honestly this gets thrown around after every Paradox release (and most other games I buy, but I buy alot of Paradox games).

What you get with Paradox is a commitment to continuous improvement - they don't walk away from their games. Like it or loathe it their track record is similar to the big OS providers. They occasionally screw up with a big patch / update release and have to scramble to correct it especially for major changes. Hell there are things in Microsoft Office which still drive me nuts 15 years after I first noticed them. I don't get me on to Android releases breaking perfectly good apps.

Stay with the program and you get a quality product - eventually. Just like Windows

A commitment to continous improvement take QA to deliver on. It is also a somewhat puzzling reflection over mitigating factors to the fundamentally unethical business practice of delivering incomplete products.

Their track record indicates to me that they will continue to spawn the DLC. Fixes and improvements are are funded by the DLC and will always be secondary to producing new content.

It is OK to accept this and pay the company if you like the games despite the bugs (like i do). But don't mistake their business principles for some kind of fairy story. The community pays for this with real money.

 

[GAGA Extrem]

The toils of the mountains lie behind us.
Before us lie the toils of the plains.

 

[Alblaka]

The toils of the mountains lie behind us.
Before us lie the toils of the plains.

I will try my hardest not to over-think this, but I assume this implies that you consider these kind of bugs, as blatantly obvious as they are, irrelevant in severity compared to the bugs you actually found prior to release?

A fair sentiment, I suppose.

 

[Alex_brunius]

Something else to consider is that when testing complex software any change to the code brings with it:

1. High chance of regressions ( bugs fixed previously popping back up )
2. High chance of new bugs that are totally unrelated to the function that was coded

This means that as long as someone is making changes to the code even just fixing bugs, other things that have tested OK and worked perfectly fine 50 times before can suddenly break in the last code deployment the final day 1 patch that is the only one you see and play.

Saying that Paradox don't have "intention/budget to fund a proper QA" is easy to prove wrong since they have publicly posted the people that work at this department, including recruitment adds: http://career.paradoxplaza.com/departments/quality-assurance

I also managed to dig up some evidence someone on reddit found before the Stellaris release that a closed Beta had already been running for at least 5 months prior:

http://i.imgur.com/GD0fYOK.png
( They wouldn't need to distribute it via Steam before release if they only ran internal Beta )

 

[Lord Milutin]

I fully support this idea. I found over 20 bugs in just 3 days and they are really evident. Testing things and doing bug reports is something I really like, so a program like this would certainly benefit us and them.