Fermin here. Danny, I loved working with you at Google but let me correct you about my promotion and money. Not true.
I respect your point of view around our technology. You may like it or not (some folks love it), but please do not make statements about me you do not really know :)
And to be clear, I believe this technology makes security researchers scale on different aspects. At least I had first hand experience with this and our goal is to make security easy for non security folks... this technology enables us to do this.
Danny, I am the CEO and founder of Semmle. I will refrain from arguing about the value of our product and technology. However, I must correct your statement about Fermín which is utterly false. He took a huge pay cut to come to Semmle. Please stick to facts when talking about people.
So I did read a whitepaper about static analysis at Google, and how it was largely self-serve - let developers run the tools and fix what it tells them to as they see fit. I’m wondering if it was under this model where you found it was not useful. I would not expect it to provide much value in that scenario, and would not be surprised by your feedback.
If your data is closer to a model where security bug hunters whose sole job is to find vulnerabilities and audit code, and it was deemed not useful in that scenario, then yes, I am at odds with your claim. Admittedly, that’s a pretty niche set of customers. If you don’t learn Semmle QL, and you aren’t writing queries, it’s probably not for you.
Its run in both self-service (output to developers), guided (output to product security oncall of security engineers) and used ad-hoc to power up manual security reviews. Depending on the accuracy of each rule and the impact of the pattern of security flaw the rule finds it is promoted to ultimately output to developers directly.
It finds about a third of the security vulns we unearth each year.
That’s been my approach as well. An astonishingly large number of companies think they can buy an off the shelf static analysis tool and pipe the default output to developers. That’s counterproductive. A very small percentage of developers will understand the output, be able to assess the exploitability/severity, and care about fixing it. One might think you could then just have them take the “better safe than sorry” approach and fix everything, but FP rates for all of the commercial tools make that completely untenable. At the same time, you can’t expect to convince small teams of developers to model everything out and define sources/sinks using some obscure DSL that they have to learn. But, there are classes of issues that are extremely high impact, but only low accuracy static analysis rules can find the candidates. It’s that part in the middle that you don’t want to throw out, but you need security experts to vet. Other cases with high confidence checks are appropriate to short circuit straight to the devs, but it’s a bad first step.
Annoyingly, now the GP post is now deleted, the context to my comment looks different, and I can't delete my comment.
Since the GP has been deleted I'll respect that and not reference specifics, but I want to clarify for any passers by, much of the GP comment I replied to was of a detailed and technical nature, about things like performance enhancements, features and semantic analysis approaches that could make the tools useful in more use-cases - very different from the rather general and personal criticisms I see elsewhere in the nearby comment tree.
It's the suggested technical and product enhancements that I felt was potentially useful feedback, rather than any of the criticism (I can understand why those are deleted).
Comments
"and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it."
Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :)
I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation.
IE it's not the kind of offer that really required a lot of faith
I'll try to write a bit more later after i think about how to frame it :)
Fermin here. Danny, I loved working with you at Google but let me correct you about my promotion and money. Not true.
I respect your point of view around our technology. You may like it or not (some folks love it), but please do not make statements about me you do not really know :)
And to be clear, I believe this technology makes security researchers scale on different aspects. At least I had first hand experience with this and our goal is to make security easy for non security folks... this technology enables us to do this.
Happy to sync in private over a coffee!
[deleted]
All good Danny, we had good times at Google... let's remember those :)
Coffee offer is still there!
Danny, I am the CEO and founder of Semmle. I will refrain from arguing about the value of our product and technology. However, I must correct your statement about Fermín which is utterly false. He took a huge pay cut to come to Semmle. Please stick to facts when talking about people.
So I did read a whitepaper about static analysis at Google, and how it was largely self-serve - let developers run the tools and fix what it tells them to as they see fit. I’m wondering if it was under this model where you found it was not useful. I would not expect it to provide much value in that scenario, and would not be surprised by your feedback.
If your data is closer to a model where security bug hunters whose sole job is to find vulnerabilities and audit code, and it was deemed not useful in that scenario, then yes, I am at odds with your claim. Admittedly, that’s a pretty niche set of customers. If you don’t learn Semmle QL, and you aren’t writing queries, it’s probably not for you.
Here is our experience building and using program analysis as part of our product security efforts at facebook: https://engineering.fb.com/security/zoncolan/.
Its run in both self-service (output to developers), guided (output to product security oncall of security engineers) and used ad-hoc to power up manual security reviews. Depending on the accuracy of each rule and the impact of the pattern of security flaw the rule finds it is promoted to ultimately output to developers directly.
It finds about a third of the security vulns we unearth each year.
That’s been my approach as well. An astonishingly large number of companies think they can buy an off the shelf static analysis tool and pipe the default output to developers. That’s counterproductive. A very small percentage of developers will understand the output, be able to assess the exploitability/severity, and care about fixing it. One might think you could then just have them take the “better safe than sorry” approach and fix everything, but FP rates for all of the commercial tools make that completely untenable. At the same time, you can’t expect to convince small teams of developers to model everything out and define sources/sinks using some obscure DSL that they have to learn. But, there are classes of issues that are extremely high impact, but only low accuracy static analysis rules can find the candidates. It’s that part in the middle that you don’t want to throw out, but you need security experts to vet. Other cases with high confidence checks are appropriate to short circuit straight to the devs, but it’s a bad first step.
[deleted]
I think there's some great feedback here, for anyone at Semmle thinking about how to develop the tool further.
Annoyingly, now the GP post is now deleted, the context to my comment looks different, and I can't delete my comment.
Since the GP has been deleted I'll respect that and not reference specifics, but I want to clarify for any passers by, much of the GP comment I replied to was of a detailed and technical nature, about things like performance enhancements, features and semantic analysis approaches that could make the tools useful in more use-cases - very different from the rather general and personal criticisms I see elsewhere in the nearby comment tree.
It's the suggested technical and product enhancements that I felt was potentially useful feedback, rather than any of the criticism (I can understand why those are deleted).