I saw thesis about "hey, this game was tested by a lot of beta testers! It was properly tested" some times now, so I believe I should explain my position about "Paradox, please hire more testers". It's kind of sharing experience - I have 10 years working in soft and game development, QA position too.
Beta test can never replace QA testing. Why? Because it's different things.
What is testing style in beta test? Well, you are taking some players, giving them your game and they are starting playing. To be honest, it's normal game, just this people looks for bugs with more attention and tends not to just ignore bugs (that's what often have place in regular playing). I don't want to say it's for nothing, not at all. This way you can find fatal bugs as "game crushes every time technology screen opens", for example. Or, not the least issue, it's kind of balance testing - players looks for balance, so you can collect this statistics without forum rage or focus groups. It's good practice (and cheap, I should notice).
But players tend to ignore issues that not visible for them with their normal play-style. They will not notice, for example, that bug with AI undespawnable levies - they haven't a habit to count active units AI have just now. Or, for example, not many players checks AI buildings and noticing that AI don't build anything or build too slow as bug, especially solvable bug. AI is Artificial Idiot after all, right?
To notice such things you should not play a game. Even if you're playing bugged game that you should help to do better, it's not enough. You should do your job, often boring, often unresulted. You shouldn't look aside to your main job. Testing issues such "is AI making wars normally?" takes 8 man-hours. Maybe more. Maybe less, but not often, because there is can be a lot of bugs here, different and obscure bugs. And you should have some experience to look such places player never will. You should see a game from developer, not player PoV. And if you can't do such things, you'll lose your job.
That's why beta test, even great, big successful beta test, not make release "well-tested".
Beta test can never replace QA testing. Why? Because it's different things.
What is testing style in beta test? Well, you are taking some players, giving them your game and they are starting playing. To be honest, it's normal game, just this people looks for bugs with more attention and tends not to just ignore bugs (that's what often have place in regular playing). I don't want to say it's for nothing, not at all. This way you can find fatal bugs as "game crushes every time technology screen opens", for example. Or, not the least issue, it's kind of balance testing - players looks for balance, so you can collect this statistics without forum rage or focus groups. It's good practice (and cheap, I should notice).
But players tend to ignore issues that not visible for them with their normal play-style. They will not notice, for example, that bug with AI undespawnable levies - they haven't a habit to count active units AI have just now. Or, for example, not many players checks AI buildings and noticing that AI don't build anything or build too slow as bug, especially solvable bug. AI is Artificial Idiot after all, right?
To notice such things you should not play a game. Even if you're playing bugged game that you should help to do better, it's not enough. You should do your job, often boring, often unresulted. You shouldn't look aside to your main job. Testing issues such "is AI making wars normally?" takes 8 man-hours. Maybe more. Maybe less, but not often, because there is can be a lot of bugs here, different and obscure bugs. And you should have some experience to look such places player never will. You should see a game from developer, not player PoV. And if you can't do such things, you'll lose your job.
That's why beta test, even great, big successful beta test, not make release "well-tested".