I want to start my post with a today's screenshot. It doesn't really form an argument, but it is emblematic of the state of PDS/PDX releases of the last decade.
I think that PDX should reconsider their stance on QA's work. In addition to frustrated ramble I actually have a more or less specific suggestion on what improvements should be made for some of the problems to never reappear. I think that while the PDX'es semi-monopolistic position has so far prevented the true GSG competition from appearing, the quality fell dramatically enough that it's only a matter of time before PDX loses its edge, unless the situation changes. I hope that Project Tinto is in the stage of development where the suggestion can still be implemented.
Some disclaimers:
1) by my work, I'm not a QA professional, and while I interact with these sometimes from a business side (i.e. I need some work do be done by development, and after it's done, QA sometimes works with me to ensure that the end result fits), I am not at all an expert in the field
2) I'm absolutely not blaming any specific QA guys or even teams. I tried discussing this on the V3 Discord, and got very angry answers from usually very friendly PDS team members. The answers were along the lines of "it's not really QA job, they just flag the problems they find" and "the guys are doing their best, given the circumstances". However, those are not on point. I'm unhappy, and I believe, objectively unhappy with the quality in most senses. In other words, in a lot of the latest PDX releases, quality is not assured. If QA teams in PDX are not responsible for that, then it's the question for Johan and game directors, "what can be done about it"
So, apart from the very humble ask to just invest more in releasing the games and their updates more polished, I actually have one specific thing I would want to discuss.
That is the emphasis on simulation testing and metrics-measured approach to problem finding.
While I know that some "overnight testing" is done, I believe that the importance of it is strongly underestimated by the devs of all games. I think that massive playerless simulation should be the main tool for testing the builds that don't actually CTD on day 1.
I think that a combination of initial testing and expert opinion should be done to determine non-anomalous distributions of a lot of metrics over multiple playthroughs:
"Number of wars declared during the game", "number of wars won by the attacker", "sum of construction ability of top-10 countries in exactly the middle of the game", "number of people migrated to the New World over the course of the game", "number of buildings X built by all nations for each decade", "number of alliances formed, honoured and broken", "rules with trait X appearing" etc, literally thousand of metrics.
Then all new builds should be sent to playerless testing to get literally hundreds of results. Anomalies in the metrics in the new builds should give alerts that a QA person will then check, depending on the alert severity.
This approach will prevent shipping patches with problems like:
1) After year 10, the game inevitably CTDs because of a broken event for some country on the edge of the world firing after its leader changes
2) Characters are immortal when they shouldn't be
3) AI never declares war
4) AI stops building
5) Migration brakes or works only under very rare circumstances
This (buying 100 high-end PCs and waiting some time for each build) doesn't seem very expensive compared to the severity of the current problems or the labour cost of QA, and I really don't understand why this approach hasn't been implemented yet.
I really want the PDX games to get better, and I hope that this thread will at least spur a discussion on what can be done to avoid disasters in release builds.
I think that PDX should reconsider their stance on QA's work. In addition to frustrated ramble I actually have a more or less specific suggestion on what improvements should be made for some of the problems to never reappear. I think that while the PDX'es semi-monopolistic position has so far prevented the true GSG competition from appearing, the quality fell dramatically enough that it's only a matter of time before PDX loses its edge, unless the situation changes. I hope that Project Tinto is in the stage of development where the suggestion can still be implemented.
Some disclaimers:
1) by my work, I'm not a QA professional, and while I interact with these sometimes from a business side (i.e. I need some work do be done by development, and after it's done, QA sometimes works with me to ensure that the end result fits), I am not at all an expert in the field
2) I'm absolutely not blaming any specific QA guys or even teams. I tried discussing this on the V3 Discord, and got very angry answers from usually very friendly PDS team members. The answers were along the lines of "it's not really QA job, they just flag the problems they find" and "the guys are doing their best, given the circumstances". However, those are not on point. I'm unhappy, and I believe, objectively unhappy with the quality in most senses. In other words, in a lot of the latest PDX releases, quality is not assured. If QA teams in PDX are not responsible for that, then it's the question for Johan and game directors, "what can be done about it"
So, apart from the very humble ask to just invest more in releasing the games and their updates more polished, I actually have one specific thing I would want to discuss.
That is the emphasis on simulation testing and metrics-measured approach to problem finding.
While I know that some "overnight testing" is done, I believe that the importance of it is strongly underestimated by the devs of all games. I think that massive playerless simulation should be the main tool for testing the builds that don't actually CTD on day 1.
I think that a combination of initial testing and expert opinion should be done to determine non-anomalous distributions of a lot of metrics over multiple playthroughs:
"Number of wars declared during the game", "number of wars won by the attacker", "sum of construction ability of top-10 countries in exactly the middle of the game", "number of people migrated to the New World over the course of the game", "number of buildings X built by all nations for each decade", "number of alliances formed, honoured and broken", "rules with trait X appearing" etc, literally thousand of metrics.
Then all new builds should be sent to playerless testing to get literally hundreds of results. Anomalies in the metrics in the new builds should give alerts that a QA person will then check, depending on the alert severity.
This approach will prevent shipping patches with problems like:
1) After year 10, the game inevitably CTDs because of a broken event for some country on the edge of the world firing after its leader changes
2) Characters are immortal when they shouldn't be
3) AI never declares war
4) AI stops building
5) Migration brakes or works only under very rare circumstances
This (buying 100 high-end PCs and waiting some time for each build) doesn't seem very expensive compared to the severity of the current problems or the labour cost of QA, and I really don't understand why this approach hasn't been implemented yet.
I really want the PDX games to get better, and I hope that this thread will at least spur a discussion on what can be done to avoid disasters in release builds.
Last edited:
- 4
- 2
- 1
