I've seen coworkers run semmle queries across the entire Windows OS codebase and find hundreds of issues which were/could result in security vulnerabilities. They've also leveraged it for variant analysis. If I'm not mistaken, the security teams are the largest internal users of Semmle at Microsoft.
You're right though, it's not a panacea, and it could probably be great for other uses too.
If you use the out of the box rules for any of these tools the false positive rate will usually be pretty high. The trick is to write custom rules that are more tailored to your code.
It took me a few months to get decent results with a low false positive rate. We haven’t had the tooling in place long enough to give hard stats but our aim is to have a false positive rate of less than 25%. Another great thing that these tools provide (if the results are valid) is that they let more junior members of the team/developers not as familiar with security issues to be able to understand the vulnerabilities found as they display a nice call flow graph/diagram that’s shows source to sink.
Comments
I've seen coworkers run semmle queries across the entire Windows OS codebase and find hundreds of issues which were/could result in security vulnerabilities. They've also leveraged it for variant analysis. If I'm not mistaken, the security teams are the largest internal users of Semmle at Microsoft.
You're right though, it's not a panacea, and it could probably be great for other uses too.
Any idea what the false positive rate is?
If you use the out of the box rules for any of these tools the false positive rate will usually be pretty high. The trick is to write custom rules that are more tailored to your code.
So how much is involved in writing the rules, and at the end of it, what is the net false positive?
It took me a few months to get decent results with a low false positive rate. We haven’t had the tooling in place long enough to give hard stats but our aim is to have a false positive rate of less than 25%. Another great thing that these tools provide (if the results are valid) is that they let more junior members of the team/developers not as familiar with security issues to be able to understand the vulnerabilities found as they display a nice call flow graph/diagram that’s shows source to sink.