I have a really crazy idea on how you could test the GUI in a way that would both save time and provide something more valuable than just testing true == true.
My idea is to create tests that take screenshots and do a diff of them over time. You then could set a change % threshold that would signal a test "failure" signal. Your QA team could then run through that process and see that something significant changed. Maybe that is fine, but maybe that is hugely unintended.
Having a Time Machine of screenshots of different processes, you could compare changes easily and see if they are worth further investigation. For example, this would be useful if you change some CSS or JS for just one page, and it ends up breaking another page.
The key point of this system is not that it would tell when your system is broken, but rather that there was a significant change that occurred that might have broken something. It's not a substitute for human analysis or thought.
Is anyone doing something like this and would it be useful to anyone else?
My company's QA team set up a system like that around 1999. There were far too many false-positives, because the GUI intentionally changes all of the time during development. So instead of testing functionality, the team spent all of their time updating screenshots. The worst was when we made a very simple style change that affected every page in the application; they'd have to redo every single screenshot instead of just doing a 5-second test that results in "Yeah, the banner is the right shade of blue now, and I know it's used on every page."
Dunno why your comment is downvoted, this is a valuable testing strategy. Taking screenshots as an automated tool walks through your GUI is a great way to find regressions without writing tests for every last pixel. And you can run something like this on it's own or bolted onto existing test cases.
Facebook does this with a project called Huxley[1]. It seems cool, but these sorts of tools have always suffered from the problem of brittle tests, so it is not a silver bullet. It does seem that it would work well, however, in systems that require stringent oversight around UI changes (like facebook). Places where "if a piece of UI changes by a pixel, we want to know about it and OK the change" is the standard (most apps do not fall into this category though).
I actually did pretty much exactly this for a previous job for testing a rendering engine - we had a 'golden master' set of screenshots, a bunch of code to render those original golden masters, then used perceptual diff (http://pdiff.sourceforge.net) to check against the golden masters.
It was a bit of an experiment and didn't get used that much - though it did come in handy when trying to write UI rendering that worked with XAML
There are tools that do something similar. At Siemens we used a tool called T-Plan (t-plan.com) to test behavior of a railroad control system. It wasn't perfect, but it worked surprisingly well. eggplant (http://www.testplant.com/eggplant/testing-tools/) also looked good, but didn't happen to work for our system at the time (not really a comment on eggPlant).
Comments
I have a really crazy idea on how you could test the GUI in a way that would both save time and provide something more valuable than just testing true == true.
My idea is to create tests that take screenshots and do a diff of them over time. You then could set a change % threshold that would signal a test "failure" signal. Your QA team could then run through that process and see that something significant changed. Maybe that is fine, but maybe that is hugely unintended.
Having a Time Machine of screenshots of different processes, you could compare changes easily and see if they are worth further investigation. For example, this would be useful if you change some CSS or JS for just one page, and it ends up breaking another page.
The key point of this system is not that it would tell when your system is broken, but rather that there was a significant change that occurred that might have broken something. It's not a substitute for human analysis or thought.
Is anyone doing something like this and would it be useful to anyone else?
My company's QA team set up a system like that around 1999. There were far too many false-positives, because the GUI intentionally changes all of the time during development. So instead of testing functionality, the team spent all of their time updating screenshots. The worst was when we made a very simple style change that affected every page in the application; they'd have to redo every single screenshot instead of just doing a 5-second test that results in "Yeah, the banner is the right shade of blue now, and I know it's used on every page."
Dunno why your comment is downvoted, this is a valuable testing strategy. Taking screenshots as an automated tool walks through your GUI is a great way to find regressions without writing tests for every last pixel. And you can run something like this on it's own or bolted onto existing test cases.
Facebook does this with a project called Huxley[1]. It seems cool, but these sorts of tools have always suffered from the problem of brittle tests, so it is not a silver bullet. It does seem that it would work well, however, in systems that require stringent oversight around UI changes (like facebook). Places where "if a piece of UI changes by a pixel, we want to know about it and OK the change" is the standard (most apps do not fall into this category though).
[1] https://github.com/facebook/huxley
I actually did pretty much exactly this for a previous job for testing a rendering engine - we had a 'golden master' set of screenshots, a bunch of code to render those original golden masters, then used perceptual diff (http://pdiff.sourceforge.net) to check against the golden masters.
It was a bit of an experiment and didn't get used that much - though it did come in handy when trying to write UI rendering that worked with XAML
There are tools that do something similar. At Siemens we used a tool called T-Plan (t-plan.com) to test behavior of a railroad control system. It wasn't perfect, but it worked surprisingly well. eggplant (http://www.testplant.com/eggplant/testing-tools/) also looked good, but didn't happen to work for our system at the time (not really a comment on eggPlant).
Houdini [1] aimed to do this, but it's been a while since I checked it out -- not sure if it launched or is still under development.
[1] http://www.tryhoudini.com
You mean something like this? https://dpxdt-test.appspot.com/