Most of their tests take half a second each to run? Some examples:
"The first story displays an image and not an alt text attr if one is defined. (793ms)"
"The module banner color is BBC News Red (552ms)"
The 13 tests shown in the screenshot take an average of 577ms each, a total of over 7.5 seconds. 13 simple tests with wild variance in execution time. Checking a module banner color? 42ms. Checking a module background color? 552ms. So a 12x increase for checking a different color within a module?
Those tests are going to rot because of the expense of executing them and they'll be discarded. Over a 500ms average per test just isn't sustainable, particularly at the scale required for checking every conceivable kind of background color.
I work on a rails codebase with some tests that take 20 seconds to run. The entire suite is normally 5 minutes. I made a ruby gem to take the entire thing down to 20 seconds, which is much better for flow.
But I agree those are definitely smelly tests with questionable value return, and would not be surprised if they do indeed rot.
Comments
Most of their tests take half a second each to run? Some examples: "The first story displays an image and not an alt text attr if one is defined. (793ms)" "The module banner color is BBC News Red (552ms)"
The 13 tests shown in the screenshot take an average of 577ms each, a total of over 7.5 seconds. 13 simple tests with wild variance in execution time. Checking a module banner color? 42ms. Checking a module background color? 552ms. So a 12x increase for checking a different color within a module?
Those tests are going to rot because of the expense of executing them and they'll be discarded. Over a 500ms average per test just isn't sustainable, particularly at the scale required for checking every conceivable kind of background color.
I work on a rails codebase with some tests that take 20 seconds to run. The entire suite is normally 5 minutes. I made a ruby gem to take the entire thing down to 20 seconds, which is much better for flow.
But I agree those are definitely smelly tests with questionable value return, and would not be surprised if they do indeed rot.