I find the comparison of software development and surgery/flying an air plane very lacking. It restricts the term "Software" in the post title to very critical systems (OpenSSL admittedly being one of them).
The author proposes automated tests/Software "checklists" as the solution to all problems. The problem with automated tests is that they can only catch "expected" bugs. The author describes an idea for an automated test that _might_ have caught the Heartbleed bug. Big deal, now that we know all about the bug. A next unforeseen type of vulnerability will come, and no automated test will catch it.
I think the article greatly exaggerates the applicability and capabilities of automated tests (or "check lists"). I'm a big fan of all things TDD and consider myself a very rigorous programmer (surely often to the detriment of speed/pragmaticism). But I don't need a check list of the complexity of a flight check to describe the rules I use in my work.
A next unforeseen type of vulnerability will come, and no automated test will catch it.
But Heartbleed wasn't a new type of vulnerability. It's the same type of vulnerability we've known about at least since the dawn of C. Almost all bugs are of known types. How often do we see new types of vulnerabilities? Maybe a couple a year. If we had only a couple vulnerabilities a year across all software, we'd be many orders of magnitude better off.
Sure, I'm not saying the proposed automated test is bad. I think it might be sensible to add it. As I said, I'm 100% for automated tests (I should be, as the co-founder of a startup specializing in test automation tools). But ascribing magical capabilities to catch all future bugs to automated tests / "check lists" is IMHO detrimental to the cause because it leads to too high expectations.
A lot of stuff that gets on checklists is because not knowing about it killed someone. Other documented procedures came from time in the simulator - somewhat anagolous to manual user testing.
To be honest, the only people I know ascribing "magical capabilities" to unit testing are those disparaging it. The rest of us think it's great, and makes life a lot easier, but we've all been bitten by that unexpected bug when you see it working in the real world.
Exactly. I've always hated the "You can't test what you don't know so it's worthless." Well sorry, I can test what I do know and make sure that I don't break that. As new errors come upon, you add it to your tests (akin to checklists, essentially), and make sure you don't break that use case again.
Except that lots of errors are already known, and get repeated anyhow. Saying that we can't guard against unforeseen errors is not quite the point.
Errors we already know about are known because they are common. Preventing those errors is a net win, regardless of whether unforeseen errors will occur.
Comments
I find the comparison of software development and surgery/flying an air plane very lacking. It restricts the term "Software" in the post title to very critical systems (OpenSSL admittedly being one of them).
The author proposes automated tests/Software "checklists" as the solution to all problems. The problem with automated tests is that they can only catch "expected" bugs. The author describes an idea for an automated test that _might_ have caught the Heartbleed bug. Big deal, now that we know all about the bug. A next unforeseen type of vulnerability will come, and no automated test will catch it.
I think the article greatly exaggerates the applicability and capabilities of automated tests (or "check lists"). I'm a big fan of all things TDD and consider myself a very rigorous programmer (surely often to the detriment of speed/pragmaticism). But I don't need a check list of the complexity of a flight check to describe the rules I use in my work.
But Heartbleed wasn't a new type of vulnerability. It's the same type of vulnerability we've known about at least since the dawn of C. Almost all bugs are of known types. How often do we see new types of vulnerabilities? Maybe a couple a year. If we had only a couple vulnerabilities a year across all software, we'd be many orders of magnitude better off.
Sure, I'm not saying the proposed automated test is bad. I think it might be sensible to add it. As I said, I'm 100% for automated tests (I should be, as the co-founder of a startup specializing in test automation tools). But ascribing magical capabilities to catch all future bugs to automated tests / "check lists" is IMHO detrimental to the cause because it leads to too high expectations.
They don't, and the article doesn't imply this.
A lot of stuff that gets on checklists is because not knowing about it killed someone. Other documented procedures came from time in the simulator - somewhat anagolous to manual user testing.
To be honest, the only people I know ascribing "magical capabilities" to unit testing are those disparaging it. The rest of us think it's great, and makes life a lot easier, but we've all been bitten by that unexpected bug when you see it working in the real world.
Exactly. I've always hated the "You can't test what you don't know so it's worthless." Well sorry, I can test what I do know and make sure that I don't break that. As new errors come upon, you add it to your tests (akin to checklists, essentially), and make sure you don't break that use case again.
Except that lots of errors are already known, and get repeated anyhow. Saying that we can't guard against unforeseen errors is not quite the point.
Errors we already know about are known because they are common. Preventing those errors is a net win, regardless of whether unforeseen errors will occur.
I'm with you. It's just that the article IMHO makes automated tests seem like a cure for every bug (future or present) in the world.
I also agree that it makes sense to try and avoid known errors. And that automated tests are often a great tool for achieving this.