Ive seen it done on a few other games in development, essentially forum members here who are willing to purchase in advance a copy of victoria II and recieve patches during the beta test in return for providing feedback, bug reports etc at least stage in the process be able to buy in an advance copy get a buggy game for the purposes of getting twice weekly patches etc via the forums in return we get to play bug test etc vic II before its realise. It could help spot bugs quicker and provide later development feedback for paradox. COuld create a subforum for payed in advance beta testers who would also recieve the final copy etc. Each patch version a dev would start a thread for that version number we could downlaod patch play report back. And so forth. any thoughts bad idea, good idea, to late to suggest not gunna happen , will it happen etc ?
Idea for paradox re beta testing available to all interested
- Thread starter ollobrains
- Start date
-
We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.
PI have used beta-testers previously on their titles, and I believe they will again. Also this time they are sending the game to a professional testing company, to help weed out most of the bugs, crashes etc. before launch.
I don't know anything about the subject (I'm sure Johan can better inform us about it), but allowing the beta-testing for a few dozen people who pay for it and continually compiling and releasing beta patches to the net for the said beta-testers to upgrade takes some worktime (Not sure how it is done, nor how the upgrading of the QA beta games are also done), but since they are already using QA Studios, it seems a bit of wasted development time, since my feedback will surely come to professional QA studios who are payed to do that anyway.
What the OP is basically suggesting is a closed beta which people pay to enter. I am not sure that would yield any better results than a closed beta where people are volunteers but haven't paid for their test version of the game.
Exactly. Usually the people who enter these betas are hardcore paradox fanboys and as we saw with all the hype surrounding HOI3 before its release they were just so stoked about being able to play the game that they did very little actual bug finding. I honestly don't know how the testers could miss the dozens of game ruining bugs unless they purposefully didn't care about them.
It's a shame too, but I guess it might explain why there are no "HoI3 Beta Tester" Icons, that would put a big target on their heads.
There haven't been any such icons for the latest betas if there were i would have 2 such icons under my avatar.
Ah, I hadn't noticed that.
I suppose the idea in the OP could improve results over the pure volunteer beta but I'm optimitsic the new process Paradox is using will work out.
I suppose the idea in the OP could improve results over the pure volunteer beta but I'm optimitsic the new process Paradox is using will work out.
ive partaken in these projects just a random thoguht, fact paradox have a dedicated bug testing outfit negates this idea nayway
I am a little concerned that Johan talks about using "proper QA studios instead."
A proper QA studio will be better at finding implementation bugs - such as compatibility problems, unexpected dependencies, or CTDs.
What the probably won't find is design errors, historical errors, balance issues, and interface improvements - but a beta test will find these.
The result is that both sorts of testing are required.
Going back to the idea, one way would be to produce a "collector's edition" like the one for Rome. Make it available to selected beta testers at 1/2 price (and below RRP of the standard edition.) Beta testers only get the discount is they contribute sufficiently - be it finding bugs, confirming that there aren't bugs, by producing a suitable mod, even just some interesting AARs.
A proper QA studio will be better at finding implementation bugs - such as compatibility problems, unexpected dependencies, or CTDs.
What the probably won't find is design errors, historical errors, balance issues, and interface improvements - but a beta test will find these.
The result is that both sorts of testing are required.
Going back to the idea, one way would be to produce a "collector's edition" like the one for Rome. Make it available to selected beta testers at 1/2 price (and below RRP of the standard edition.) Beta testers only get the discount is they contribute sufficiently - be it finding bugs, confirming that there aren't bugs, by producing a suitable mod, even just some interesting AARs.
That's very unfair assumption.
In most of such projects, valid bug reports from testers are simply ignored because of the late stage and time schedule.
Similarly, some of the stuff CAN'T even be properly tested - for example it's common thing for devs to not optimize beta versions - so main issue with HoI3 that was present at the start (abysmal performance) was something that testers most likely were not even allowed to log as an issue.
Top that with very limited testing of game balance ("it's not a bug, that happened only in your particular game") and inability of amateurs to properly test the game that takes 2-3 hours to move from 1936 to 1939 on "V. Fast" speed (then it gets progressively slower) and you have your answer why HoI3 was virtually untestable.
I can even imagine that couple of people from beta team quit the whole process out of sheer frustration (and others were trying to do what they possibly could till the last moment before the release) - we most likely never know because of the nature of NDA (Non Disclosure Agreement) every tester got to sign. But stating that it was beta's fault is really unfair. Some of the less active testers surely can be blamed, but main issue was the development cycle and tests organization, not volunteers.
And this gets improved in case of Vicky 2, which is fantastic - keeping experienced members of modding community as advisors on the design, alpha stage and hiring pros for proper beta, when you polish the product sounds like a good middleground between community expectations and proper quality of the product.
First of all I want to say that I dont have any experience with game testing or programming but after being a fan of PI for years I have made an opinion on this subject.
I was surprised that PI didn't bring in pro besta testers already in EU3 because I thought the game had now gotten so big and complex that volunteer amateur beta testers would not be able to really reach in and find the bugs and help the devs to tweak the game to a much higher standard which I expected of the clausewitz engine.
Now Im not disrespecting the beta testers of Eu3 or HOI3, I think they did their best but the game was too big and complex for them. Now I hope we will see a much higher standard and games that will have a more professional look from start (PI patching policy is, in my opinion, a bit too generous and makes the games look slightly unprofessional).
I was surprised that PI didn't bring in pro besta testers already in EU3 because I thought the game had now gotten so big and complex that volunteer amateur beta testers would not be able to really reach in and find the bugs and help the devs to tweak the game to a much higher standard which I expected of the clausewitz engine.
Now Im not disrespecting the beta testers of Eu3 or HOI3, I think they did their best but the game was too big and complex for them. Now I hope we will see a much higher standard and games that will have a more professional look from start (PI patching policy is, in my opinion, a bit too generous and makes the games look slightly unprofessional).
as a side note on this thread PI already have their beta testers but ... making history II are doing exactly what i was suggesting http://www.gamersgate.com/DD-MH2/making-history-ii-the-war-of-the-world. Might be an idea for future paradox titles
I'm a professional tester (just not for games), and I can tell you that there is a WORLD of difference between user acceptance testing (equivalent to beta testing) and hiring pros. This is not to say that you can't succeed with only one, or that you're doomed if you pick one over the other...
1.) Pros are generally able to get involved earlier, partially since when you're paying someone, and they advise you to do it, you listen better than when someone you got for free tells you to do it. A rule of thumb in the software industry is that any bug costs more to fix later in the cycle than earlier. Also, by the time you've gone to beta, marketing has set a release date and wants it OUT THE DOOR.
2.) Pros understand common things to test. Users often don't. That said, betas/users have a better understanding of expectations, and will bring up things that would never have occurred to a pro.
3.) Pros will have automation tools, automation testing experience, and the like. Automation tools can tell you more than "it's slow" - often it can give you insight as to why it's slow, before the devs even have to pore over the code. Automation also lets you run many more sessions than betas possibly could. While the devs could also automate it, it's simply a different mindset between functional testers (manual), automated testers, and devs. Having all 3 types of testers is optimal from a quality/time perspective, if not always cost.
In a perfect world, Paradox would do both. However, that means they're spending extra money and time having someone deal with the goat rodeo that is a beta. Being in the middle of User Acceptance hell where I work now, I can understand completely why Paradox didn't want to do both.
Oh wait, I'm in Integration, System, and User Acceptance Hell. Three hells for the price of one!
1.) Pros are generally able to get involved earlier, partially since when you're paying someone, and they advise you to do it, you listen better than when someone you got for free tells you to do it. A rule of thumb in the software industry is that any bug costs more to fix later in the cycle than earlier. Also, by the time you've gone to beta, marketing has set a release date and wants it OUT THE DOOR.
2.) Pros understand common things to test. Users often don't. That said, betas/users have a better understanding of expectations, and will bring up things that would never have occurred to a pro.
3.) Pros will have automation tools, automation testing experience, and the like. Automation tools can tell you more than "it's slow" - often it can give you insight as to why it's slow, before the devs even have to pore over the code. Automation also lets you run many more sessions than betas possibly could. While the devs could also automate it, it's simply a different mindset between functional testers (manual), automated testers, and devs. Having all 3 types of testers is optimal from a quality/time perspective, if not always cost.
In a perfect world, Paradox would do both. However, that means they're spending extra money and time having someone deal with the goat rodeo that is a beta. Being in the middle of User Acceptance hell where I work now, I can understand completely why Paradox didn't want to do both.
Oh wait, I'm in Integration, System, and User Acceptance Hell. Three hells for the price of one!
Using pros doesn't rule out (the need for) User Acceptance/Beta tests
Especially when you outsource the bug-finding to an external company, who may not know anything about how the game is supposed to play.
The pros will be very good at finding CTDs, graphical glitches, spelling errors, non-functional keys, icons etc. ... but won't be much good on actual gameplay issues.
PI could still gain alot of valuable intel from the "Insanely good active beta-testers" .... beta testers who know PI games and have a passion for it, could help balance the game, suggest changes or improvements that might be ready for the first patch if not the actual release and in general make the product more well-rounded.
Especially when you outsource the bug-finding to an external company, who may not know anything about how the game is supposed to play.
The pros will be very good at finding CTDs, graphical glitches, spelling errors, non-functional keys, icons etc. ... but won't be much good on actual gameplay issues.
PI could still gain alot of valuable intel from the "Insanely good active beta-testers" .... beta testers who know PI games and have a passion for it, could help balance the game, suggest changes or improvements that might be ready for the first patch if not the actual release and in general make the product more well-rounded.
Beta-access for pre-orders is really just a scheme to get more people to pre-order. Their contribution to the actual testing is rather minimal.
Still, this doesn't necessarily mean it's a bad idea. If P-dox thinks this would benefit them financially, I'm all for it. They'll have to take into consideration though that this could lead to early leaks of information and even software.
Still, this doesn't necessarily mean it's a bad idea. If P-dox thinks this would benefit them financially, I'm all for it. They'll have to take into consideration though that this could lead to early leaks of information and even software.
Up till now, all beta testers were subject to NDA agreements, that stopped them from commenting on the beta content and whole testing process.
Having beta that is not NDA limited (and I can't imagine how can you expect people to sign NDA when they pay for beta) will result in hordes of angry fans sharing their angst on the forums long before the premiere.
Vicky 2 would basically need to have much, much higher polish level then any of the previous Paradox titles so far to pull this out - otherwise, it will only result in lots of negativity from pre-ordering people.
And as much as I have hope for Vicky 2 (dev concepts sound really good, so their new testing policy and more time spent on balancing stage), the very nature of Paradox games makes me sceptical on giving beta access to wide audience in the same way it happens to certain MMOs and such.
Having beta that is not NDA limited (and I can't imagine how can you expect people to sign NDA when they pay for beta) will result in hordes of angry fans sharing their angst on the forums long before the premiere.
Vicky 2 would basically need to have much, much higher polish level then any of the previous Paradox titles so far to pull this out - otherwise, it will only result in lots of negativity from pre-ordering people.
And as much as I have hope for Vicky 2 (dev concepts sound really good, so their new testing policy and more time spent on balancing stage), the very nature of Paradox games makes me sceptical on giving beta access to wide audience in the same way it happens to certain MMOs and such.
Using pros doesn't rule out (the need for) User Acceptance/Beta tests
Just remember that having a professional outfit will lengthen the development schedule by itself. Running betas also lengthens it, and only some of that can be done concurrently. As Johan noted - they were seeing a 10-20% activity rate from betas - makes it hard to justify the time and organization necessary to do both.