Paradox, include CB testing into your QA

[Alblaka]

Saying that Paradox don't have "intention/budget to fund a proper QA" is easy to prove wrong since they have publicly posted the people that work at this department, including recruitment adds: http://career.paradoxplaza.com/departments/quality-assurance

Note that you are referencing the entirety of the QA department, which covers ALL Paradox games.
And, even assuming that Paradox doesn't go with IT buisness standards and therefore doesn't budgets projects seperately in first place, this still doesn't tell us anything about the actual manhours invested into Stellaris itself.

Albeit I will obviously agree that the formulation should have been "intention/budget to fund a proper QA for Stellaris" to clarify it.

I also managed to dig up some evidence someone on reddit found before the Stellaris release that a closed Beta had been running for at least 5 months prior:

That's interesting and alters the question to "are they still doing Closed Beta post the initial launch?".

Thanks for the source pointers!

 

[Derek Pullem]

Note that you are referencing the entirety of the QA department, which covers ALL Paradox games.
And, even assuming that Paradox doesn't go with IT buisness standards and therefore doesn't budgets projects seperately in first place, this still doesn't tell us anything about the actual manhours invested into Stellaris itself.

Albeit I will obviously agree that the formulation should have been "intention/budget to fund a proper QA for Stellaris" to clarify it.



That's interesting and alters the question to "are they still doing Closed Beta post the initial launch?".

Thanks for the source pointers!

Unless Paradox have radically changed their policy, Closed betas are not paid. So a Closed Beta cannot be compared to a professional testing program. Contrary to popular belief there are not dozens of teenagers prepared to work to a fixed program of work for nothing 16 hours a day.

As I mentioned before - previous experience would suggest you would be lucky to retain 10-20% of the initial Closed beta testers post launch. For the Paradox betas I was involved in you tended to only get between 5-10 active testers operating after the first patch (judging from the bug reports / forum posts).

 

[Alex_brunius]

Note that you are referencing the entirety of the QA department, which covers ALL Paradox games.
And, even assuming that Paradox doesn't go with IT buisness standards and therefore doesn't budgets projects seperately in first place, this still doesn't tell us anything about the actual manhours invested into Stellaris itself.

Well you can compare the size of the QA department with the size of for example the programming department to get a good idea.

Programming has 29 people and QA has 30, so it's almost exactly 1 QA for each programmer, or one hour QA per hour programming (assuming they work equally long hours).

How much more do you think would be reasonable?

 

[Alblaka]

Unless Paradox have radically changed their policy, Closed betas are not paid. So a Closed Beta cannot be compared to a professional testing program. Contrary to popular belief there are not dozens of teenagers prepared to work to a fixed program of work for nothing 16 hours a day.

As I mentioned before - previous experience would suggest you would be lucky to retain 10-20% of the initial Closed beta testers post launch. For the Paradox betas I was involved in you tended to only get between 5-10 active testers operating after the first patch (judging from the bug reports / forum posts).

I'm legitimately surprised to hear this, but I suppose long-term motivation is an issue. And likely it's not very effective to keep recruiting new people, then filter out those unfit for the job, plus the fall-out you mentioned, and then trying to have some form of steady number of testers.
At times I forget that not everyone is obsessed with video games and used to devoting the entirety of his spare time to playing (and discussing) such...
I suppose in that light, I may have more harm with this whole thread, by ending up on a blacklist that will prevent me from becoming a Beta tester for any PDX product ever again ^^'. Then again, I didn't consider the possibility of applying for a Closed Beta I wasn't aware existed in first place (under the assumption the CB for Stellaris is still ongoing).

How much more do you think would be reasonable?

If the answer to that would be simple, IT would have one big issue less.
I.e. in my current project (buisness IT), we have 8 coders and 1.5 testers, and even working overtime, they can NOT keep up. However, despite having to do thorough tests for every feature on multiple platforms, I think 8 coders would be plain overkill.
The core issue here is that, despite being a rather hefty-budgeted buisness thing, I don't think the complexity of this project can be remotely compared to Stellaris, thus my numbers, whilst providing a raw frame of reference, cannot be actually applied to Stellaris QA.

 

[The Founder]

In the assumption that Paradox doesn't have the intention/budget to fund a proper QA, and in light of the rather obvious issues/bugs in 1.5, which a proper QA would have supposedly been able to identify easily, I politely request Wiz/whoever_else_is_in_charge to consider setting up a Closed Beta system of some form or nomination, to provide the Dev Team a voluntary and micro-budget QA-replacement by fan-base.

They did not had any more time to fix the issues.

Everything more testing would have acomplished would have been either:
a) a later realease.
b) less features in, because more time for buxfixing planned in.

Both ideas sound like terrible ideas for me.

 

[AlphaAsh]

The toils of the mountains lie behind us.
Before us lie the toils of the plains.

You should spend more time in the mountains if you want me to keep buying your games.

 

[AlphaAsh]

They did not had any more time to fix the issues.

Everything more testing would have acomplished would have been either:
a) a later realease.
b) less features in, because more time for buxfixing planned in.

Both ideas sound like terrible ideas for me.

Both ideas sound good to me.

 

[DreadLindwyrm]

Indeed.
And that's why establishing a closed beta system actually worth it's name (aka, with 2-digit player counts), would remedy the issue.

So... say you've got 100 beta testers. They can each manage 2 hours a night.
There's also 10 in house testers. They can manage 10 hours a day.

Your beta testing team then adds perhaps 1400 hours of testing a week, compared to 700 hours from the in-house team.

Anything the beta testers find has to be replicated (several times in some cases) by the in-house team, before it can be brought to the attention of the code team to then fix it.


Then the game is released, and suddenly you've got tens of thousands of people playing mutliple hour sessions. In a couple of days you can easily have in excessive of 100 000 hours of play, over tens of thousands (rather than a hundred or so) different styles and approaches of play.

Closed beta testing doesn't add that much actual volume to the QA testing process, and, of course, once a new incremental beta is released, it can render all previous information irrelevant as new bugs can arise in previously tested or previously untouched areas. It helps, but it's certainly no magic bullet.

 

[The Founder]

Both ideas sound good to me.

I have been hyped for 1.5. They still sound shitty to me after it has been out for 4 days. That is why I will keep choosing "Respectfully Disagree" to everyone asking for even more testing.

There is no bug I know off that a quick patch or some modding can not fix.

I wanted to get my hands on that shiny new Ethics Attraction/Faction System. Everything else was "nice to have, but also delays when it will be released".

 

[Slarkon]

What PDX needs is an upgrade to the QA/bug-fixing system so that game crushing bugs don't show up at release day. It's obvious that the "Embrace faction unhappiness" bug, which comes up every time you embrace a faction, would have made itself present, if someone had spent more than three hours testing the game. So, the conclusion is that either someone should get fired for incompetency, or they're not doing a great job overall because of understaffing or problems of an organizational nature.

idont know how many beta tests or intenral games where had but it was not enough
https://forum.paradoxplaza.com/foru...-still-go-to-habitats-in-core-worlds.1011845/

 

[DreadLindwyrm]

You should spend more time in the mountains if you want me to keep buying your games.

Both ideas sound good to me.

So what should have been cut?

How long should it have been delayed?

 

[FrancescoT]

I fully support this idea. I found over 20 bugs in just 3 days and they are really evident. Testing things and doing bug reports is something I really like, so a program like this would certainly benefit us and them.

Quoting GAGA from another similar thread:

Well, there are several factors that play into this:
(1) You gotta FIND the issue.
(2) You gotta FIX the issue.
(3) You gotta VERIFY that the issue is fixed.
(4) You gotta MAKE SURE that it doesn't get broken again by a later revision.

Finding and reporting a bug is only one, and probably the more easy, step in the process (and I would like to add step 1.5: you gotta reproduce the issue).

 

[InertFurry]

Hell, I can't even play. Been stuck in the middle of my starting system with a nonfunctional galaxy map since 1.5.

 

[Had a dad]

just an FYI, there will not be a petition here on this forum, nor will a link to one be allowed.

 

[FrancescoT]

Hell, I can't even play. Been stuck in the middle of my starting system with a nonfunctional galaxy map since 1.5.

If you are using a controller, remove it.

 

[mackau]

Remember that this is the game studio who put in three major endgame crises, used them heavily in their marketing campaign, aaaaaaaaaaaaaaaand two of them were completely broken.

 

[ColCarp]

I disagree with your tone, but applaud that you are submitting the bug reports.

I'm not sure though, what exact sort of CB are you proposing?

 

[thekaje]

OP, you should post your idea on Reddit and see where it goes

 

[Had a dad]

OP, you should post your idea on Reddit and see where it goes

what difference does it make, since the only ones that could make it happen are in the Pdx offices. and that is going out on a limb to think it's not already in place.

I mean, logically before you start a petition for something "new", shouldn't you know if said thing in fact doesn't exist?

 

[AlphaAsh]

There is no bug I know off that a quick patch or some modding can not fix.

One that didn't get live in the first place?

So what should have been cut?

How long should it have been delayed?

I don't work for Paradox.