When you work back through the causal chain enough, you end up having to be furious about 1. standardized architectures with many compatible hardware models; and 2. non hard-real-time operating systems.
Interestingly, in a modern environment (unlike in the 80s-00s), both of these seem like choices that could be made either way.
Re #1 — all the major OS manufacturers (Microsoft, Apple, Google) now have their own flagship devices running their OS, and could standardize their app-store certification processes to involve perf testing on their own hardware if they wanted. Especially for the mobile hardware: there's no reason the QA process for e.g. releasing a game on iOS can't involve certifying zero stutter on a given iPhone, and then restricting the game to only be playable on that iPhone or newer (the way consoles effectively work right now.)
Re #2 — there's no reason, other than inertia, that personal computer (including mobile) OSes still put every single task on the system into one big (usually oversubscribed, from a realtime perspective) scheduling bag. We could—using the hardware hypervisor built into all modern architectures—split PC OSes into two virtual machines: one for the foreground "app", and one for everything else; and give the foreground-app VM fixed minimum allocations of all the computer's resources. Then you could make real guarantees† about an app or game's performance, as long as it was said foreground app. It'd be very similar to the way some game consoles (e.g. the Wii) have separate "application processors" and "OS service processors", to ensure nothing steals time from the application.
† And those guarantees would also be requirements: if the foreground app says it needs its VM to have 5GB of RAM, you literally won't be able to run it if you don't have 5GB of free physical memory to hand it (though the OS would probably first try to OOM-kill some sleeping apps to give it that memory, like iOS does.) Much clearer than the current "this game will be really slow if your computer is more than four years old, but it's not exactly clear which part of the computer is below-spec" we have today.
"there's no reason the QA process for e.g. releasing a game on iOS can't involve certifying zero stutter on a given iPhone"
To guarantee that, testing has to go through _all_ possible game states. For almost any non-trivial game, that's infeasible.
"and then restricting the game to only be playable on that iPhone or newer"
There is no guarantee that newer hardware would be faster for every possible program execution, and even if it were, timing differences could affect game play.
There also is no guarantee that newer hardware produces the exact same results. For example, better anti-aliasing or fonts drawn at double resolution could affect hit detection.
This even isn't guaranteed on the 'same' hardware. For example, there might be C64's that don't have the bugs that this demo exploits.
If only things were that simple. A demo video can show that there is an interaction that doesn't stutter, not that there is no interaction that stutters.
I realize it's imperfect, but it would raise confidence to see a video of something that looks fairly consistent with the advertised usage that satisfies the benchmarks.
The video doesn't guarantee bug-free play, but it can give us confidence that the benchmarks are an honest test since we can see what the test looked like.
If it's a text editor and it just shows that opening a new blank file and typing a few words is fast, or a game that shows the game runs well in an empty room level, then we know the demo is dishonest. And having a "boring" video in their store page won't win them much buyers. You want all the ad content to look as good as possible, so the demo video would be best if it shows off the most impressive features of your application, which is where we'd expect performance problems.
That said, it's probably not feasible because creating a full benchmark script that also doubles as a real-world demo for an app would be too much burden to put on developers.
Comments
When you work back through the causal chain enough, you end up having to be furious about 1. standardized architectures with many compatible hardware models; and 2. non hard-real-time operating systems.
Interestingly, in a modern environment (unlike in the 80s-00s), both of these seem like choices that could be made either way.
Re #1 — all the major OS manufacturers (Microsoft, Apple, Google) now have their own flagship devices running their OS, and could standardize their app-store certification processes to involve perf testing on their own hardware if they wanted. Especially for the mobile hardware: there's no reason the QA process for e.g. releasing a game on iOS can't involve certifying zero stutter on a given iPhone, and then restricting the game to only be playable on that iPhone or newer (the way consoles effectively work right now.)
Re #2 — there's no reason, other than inertia, that personal computer (including mobile) OSes still put every single task on the system into one big (usually oversubscribed, from a realtime perspective) scheduling bag. We could—using the hardware hypervisor built into all modern architectures—split PC OSes into two virtual machines: one for the foreground "app", and one for everything else; and give the foreground-app VM fixed minimum allocations of all the computer's resources. Then you could make real guarantees† about an app or game's performance, as long as it was said foreground app. It'd be very similar to the way some game consoles (e.g. the Wii) have separate "application processors" and "OS service processors", to ensure nothing steals time from the application.
† And those guarantees would also be requirements: if the foreground app says it needs its VM to have 5GB of RAM, you literally won't be able to run it if you don't have 5GB of free physical memory to hand it (though the OS would probably first try to OOM-kill some sleeping apps to give it that memory, like iOS does.) Much clearer than the current "this game will be really slow if your computer is more than four years old, but it's not exactly clear which part of the computer is below-spec" we have today.
"there's no reason the QA process for e.g. releasing a game on iOS can't involve certifying zero stutter on a given iPhone"
To guarantee that, testing has to go through _all_ possible game states. For almost any non-trivial game, that's infeasible.
"and then restricting the game to only be playable on that iPhone or newer"
There is no guarantee that newer hardware would be faster for every possible program execution, and even if it were, timing differences could affect game play.
There also is no guarantee that newer hardware produces the exact same results. For example, better anti-aliasing or fonts drawn at double resolution could affect hit detection.
This even isn't guaranteed on the 'same' hardware. For example, there might be C64's that don't have the bugs that this demo exploits.
Solution: Developer provide a script that generates a demo video from the application that will be posted in the store listing.
So right there on the store website is the "demo video".
If only things were that simple. A demo video can show that there is an interaction that doesn't stutter, not that there is no interaction that stutters.
For an historical analogue: you can show hours of paying PacMan without finding its kill screen (http://www.donhodges.com/how_high_can_you_get2.htm)
I realize it's imperfect, but it would raise confidence to see a video of something that looks fairly consistent with the advertised usage that satisfies the benchmarks.
The video doesn't guarantee bug-free play, but it can give us confidence that the benchmarks are an honest test since we can see what the test looked like.
If it's a text editor and it just shows that opening a new blank file and typing a few words is fast, or a game that shows the game runs well in an empty room level, then we know the demo is dishonest. And having a "boring" video in their store page won't win them much buyers. You want all the ad content to look as good as possible, so the demo video would be best if it shows off the most impressive features of your application, which is where we'd expect performance problems.
That said, it's probably not feasible because creating a full benchmark script that also doubles as a real-world demo for an app would be too much burden to put on developers.
Re Re #2, isn't that what Microsoft has promised to do? I can't pull up articles right now, but I will if I remember.
I believe Xbox One does this. There is a hypervisor, and different VMs for the home screen and the currently active game.