Idea for paradox re beta testing available to all interested

[unmerged(71032)]

As Johan noted - they were seeing a 10-20% activity rate from betas - makes it hard to justify the time and organization necessary to do both.

Nah, it only means that SOME betas were useless, while others were working their a... off to make game playable.

As with every volunteer activity, some people got wrong expectations while applying - for some unknown reason, people actually apply to Paradox game beta assuming they will be able to play the game earlier. Instead, they get hell bugged prototype they are supposed to test - and test according to the schedule, ignoring the bugs that are tested by other teams and focusing on not on the part they want, but the one devs direct them at. So even if people apply to test, all the sudden they find out that they are supposed to test things they don't want to - and that they are here not to design stuff or make suggestions, but to spot bugs in other people designs.

As you can imagine, some people can find such "boon" not much fun and promptly leave.

 

[EGaffney]

Fans like to complain that older Paradox games were poor upon release. But fans also want to raise their status by advocating amateur rather than professional testing. Surely there's a causative connection between the two; fans can improve game quality by switching support to professional testing, but it means admitting that fans aren't best placed to test.

 

[unmerged(71032)]

I think naggy summarized it the best in one of his posts - fans are good in certain types of tests, while pros are good in different ones. Pros not neccessarily know what makes good game, while they surely know what makes working game.

Classic example for that in Paradox portfolio is EU:Rome - no one can argue that this game worked, and it worked better then any other Paradox game up to date. But was it a good game? Was it judged as good by the fanbase?

 

[TempestDK]

Just remember that having a professional outfit will lengthen the development schedule by itself. Running betas also lengthens it, and only some of that can be done concurrently. As Johan noted - they were seeing a 10-20% activity rate from betas - makes it hard to justify the time and organization necessary to do both.

Well if they cut down the betas to those 15% that were active, they would have a 100% active beta staff, and with less managing needed . Ofcourse finding the right beta testers could be an issue, unless they can still get hold of the ones that have perfomed admirably for them before.

I can understand if they feel that it is too much work for too little return. But when Johan uses phrases like "Insanely good beta-testers", I think it would be a mistake to drop the use of such (free) talent to balance the game before release.

In the end this could save them alot of game-play complaints and lots of work on the first few patches to get things balanced correctly.

Since they now will have a professional company to work on a schedule, they could let the handful of betas just use the game without having to work on specifics ... let them play around with it, checking for user-friendliness and gameplay issues. That shouldn't require too much management, and getting 20 good bug reports or suggestions in, would mean 20 less issues on release.

 

[TempestDK]

I think naggy summarized it the best in one of his posts - fans are good in certain types of tests, while pros are good in different ones. Pros not neccessarily know what makes good game, while they surely know what makes working game.

Classic example for that in Paradox portfolio is EU:Rome - no one can argue that this game worked, and it worked better then any other Paradox game up to date. But was it a good game? Was it judged as good by the fanbase?

I am not sure any amount of beta-testing would have saved Rome

 

[EGaffney]

I think naggy summarized it the best in one of his posts - fans are good in certain types of tests, while pros are good in different ones. Pros not neccessarily know what makes good game, while they surely know what makes working game.

Classic example for that in Paradox portfolio is EU:Rome - no one can argue that this game worked, and it worked better then any other Paradox game up to date. But was it a good game? Was it judged as good by the fanbase?

The "fanbase" is made of people who have been fans of the developer since before the release of a game. They are conservative about what they like in the developer's games and they are forgiving of flaws, compared to the target audience as a whole. So they will focus on the wrong things in testing, compared to professionals. Is there any evidence that amateur testers know better than professionals about what makes a "good game", or that either kind of tester actually contributed to game design (not testing) in a way that made the difference between a good and a bad game?

 

[unmerged(71032)]

The "fanbase" is made of people who have been fans of the developer since before the release of a game. They are conservative about what they like in the developer's games and they are forgiving of flaws, compared to the target audience as a whole. So they will focus on the wrong things in testing, compared to professionals. Is there any evidence that amateur testers know better than professionals about what makes a "good game", or that either kind of tester actually contributed to game design (not testing) in a way that made the difference between a good and a bad game?

What kind of evidence you want? Reports? Books written on the subject? Of course there is no evidence.

Nevertheless...

I'm not sure what do you understand by pro testing, but naggy explained fairly well, what they do. They are paid for debugging and optimization, not about finding good game mechanics, making it fit the target audience and such. Yesterday they have tested racing game, today they test Vicky 2, tomorrow - they will test pinball. They don't analize if strategy game is good, they only test if the interfaces click, if it does what manual says (in clicking terms, menu positioning and such), if game does not CTD and such.


Fan, on the other hand, is a guy that might not be IT specialist, but he is a guy that is supposed to buy the game, and got some sort of vision what such game should achieve. For pro tester, that vision is no bugs. For fan, this vision is that country X will do this, and country Y will do that, leading to something that will make him happy, because it's historical/playsible, whatever.

He can also contribute. Check the AI/event files for HoI2 and names of authors. Check scenarios data. And there was much more - that would be made in much poorer manner, if not fanbase contribution in the beta phase. If you want a proof of improving game by the fanbase, this is the closes thing I can present you.

For me, classic example (even though it seems there was no pro testing team there) of such "working well, but not so good" game is EU:Rome. It's almost absolutely bug free (compared to the other Paradox titles on the lauch day). At the same time, it was huge dissapointment to the fanbase. It might appealed to the new players, its possible. But it was one of the things that alienated the old fanbase and made people more inclined to believe that Paradox games should be bought only, when they reach bargain bin prices.

Combination of pro and fanbase testing is neccessary to keep right balance between game "working" and "being good", that's all.

 

[unmerged(75409)]

But that is not something you can fix in beta or pro testing. No amount of playtesting can make up for bad game design choices, or rushed concept development.

 

[unmerged(71032)]

But that is not something you can fix in beta or pro testing. No amount of playtesting can make up for bad game design choices, or rushed concept development.

I can't agree.

In case of Rome, it was clear right off that it got poorly implemented character managament aspect and I'm 99% sure fanbase betas (active ones) were giving feedback (if allowed) on that matter. If Rome was launched with Vae Victis elements right off, it would have different reception.

I can agree that no amount of testing will make up for poor design choices if designers choose not to listen.

 

[EGaffney]

I can't agree.

In case of Rome, it was clear right off that it got poorly implemented character managament aspect and I'm 99% sure fanbase betas (active ones) were giving feedback (if allowed) on that matter. If Rome was launched with Vae Victis elements right off, it would have different reception.

I can agree that no amount of testing will make up for poor design choices if designers choose not to listen.

But there's a difference between making a new system during design of an expansion pack, and making a new system during testing of beta versions. One shouldn't expect the designers to design, balance, and test a new feature to improve gameplay in the eyes of the small but vocal "fanbase" when everything else is ready to go. That's not how the development cycle works.

People who were content with the flaws in Paradox games in the past will overlook the reasons why Paradox was not as successful in the past.

 

[Earl Uhtred]

The "fanbase" is made of people who have been fans of the developer since before the release of a game. They are conservative about what they like in the developer's games and they are forgiving of flaws, compared to the target audience as a whole. So they will focus on the wrong things in testing, compared to professionals. Is there any evidence that amateur testers know better than professionals about what makes a "good game", or that either kind of tester actually contributed to game design (not testing) in a way that made the difference between a good and a bad game?

The fanbase can be an astonishingly destructive force

I was on Apolyton after they announced Civ 5 and some of the regulars there were in apoplexy. 'WE CAN'T HAVE HEXES! IT HAS ALWAYS BEEN SQUARES!' 'WITHOUT RELIGION CIV 5 WILL BOMB!!!' It's easy to forget the religion system was only introduced in Civ 4 and 'culture' in Civ 3 with the same tendency fighting every step of the way.

 

[Ksim3000]

I prefer Paradox to use professional beta testers. Considering the failure at launch HOI3 was, Paradox need a really good launch this time. Professional betas should help with that.

 

[unmerged(71032)]

But there's a difference between making a new system during design of an expansion pack, and making a new system during testing of beta versions.

No.

There are different types of product tests. Some of them include using test subjects from target audience to identify, if product is fulfilling their expectations and if not, what should be changed to achieve that.

Of course, you can assume that you know better what they want, launch the game... then get the feedback and deliver what they actually want in the expansion pack. This way, you risk that your game will be pruchased from the bargain bin though - and that plently of people will advertise others to shop for Paradox games this way.

Anyway, I agree that pros for testing are right decision. And we are in moot point anyway, because I also see that player/fan base, at least its most constructive part (like Clio team), is participating in the design process already.

 

[ollobrains]

fanbase can provide info feedback, historical events, possible exploits ideas etc. Professional bug testers are more atuned to finding crashes, and other weird goings on with the actual code, combination of the two like the other games out there that are being made and opening up pre order beta testing are doing but PI has made the call on victoria 2 and just hope they got it right

 

[naggy]

But that is not something you can fix in beta or pro testing. No amount of playtesting can make up for bad game design choices, or rushed concept development.

Partially true. When beta testing is scheduled at the end of the process, then generally, companies want to adhere to the schedule. If you've decided that "we beta, fix, then ship", then you are absolutely correct.

But if you do the beta, then realize that you need to take an extra month to overhaul something, and get internal support to do the overhaul, then yes, beta testing can make up for design choices. But then, the same thing goes for professional testers - plenty of projects have gone down in flames because testers noted fundamental flaws and were ignored.

The best path is pros + betas. That costs a lot of time and money, since you generally don't do those phases concurrently. When time and money are constrained, you just have to pick one (or half and half) and hope you can manage it right.

 

[Darkrenown]

It's a shame too, but I guess it might explain why there are no "HoI3 Beta Tester" Icons, that would put a big target on their heads.

We could use the HoI3 avatars without registering HoI3, but no, no icons. There weren't icons for EUIII or Rome betas either though.

 

[naggy]

The avalanche has begun. It is too late for the pebbles to vote.

 

[ollobrains]

well i assume paradox is confident in the outcome of their bug testing so now we look forward to a good quality game

 

[unmerged(149708)]

I think the issue of time and money allocated to the testing process is the most important factor of the quality of the game - rather than the beta / pros decision. If you want a quality game then you have to be prepared to delay launch and invest a lot of time and money in getting it right. Pro testers will help you pick up errors but they will not fix them. I don't think HoI 3 would have come out polished had there been pro testers - the amount of time to solve the problems that were there simply wasn't available. (AFAIK - I am not privy to the internal workings of paradox). Judging from the DDs it seems that they have placed more emphasis on this for Vic 2.

 

[ollobrains]

yeah i think they are going thro as they go, they seem to be making very good progress, guess the protesters are working bugs as they go.