I don't think so. Sophisticated threat actors have been attacking the Open Source community for decades, adding AI into the equation changes nothing from my perspective. I'd actually imagine it's harder to get the precise and perfectly-obfuscated behavior you want while going through an LLM as a midpoint.
So my guess is no. Even in the case of the XZ backdoor, it was the incredible amount of social-engineering at play that enabled the hack. The developer had been conning the repo maintainer for years, which is easier achieved with human intellect than ChatGPT and a fake Github account.
Well it is true, the XZ hack was a major social-engineering feat. Regardless, we should not be underestimating the power of these technologies. If an LLM can respond to a request to generate code, what makes you think the attacker couldn't simply add things like "Insert a buffer overflow in the code..." or what have you. And especially as this technology progresses, the capabilities will no doubt dramatically increase. Having said that, I do hope you are right!
If an LLM can respond to a request to generate code, what makes you think the attacker couldn't simply add things like "Insert a buffer overflow in the code..." or what have you.
What makes you think an attacker couldn't do that without an LLM? I'm having a hard time understanding what changes in this scenario by introducing an AI.
Pretty much anyone can write a buffer overflow exploit with enough research. The harder part is getting your patches through code review, which is probably only hindered by having ChatGPT help you. Again... the past few decades have mostly gone off without a hitch, and there have been a lot of monkeys on typewriters hooked up to the internet.
I don't know, LLM's are very good at embedding subtle changes in their responses. They are also good at researching things like "best possible place to hide bugs", etc. Is it really a stretch to imagine that these technologies won't, if not now, but "someday soon" be used in such a way? I certainly don't think so.
Humans are really good at subtlety and research. I guess my point is this:
pre-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge malicious patches and hope nobody sees them
post-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge patches that were written slightly faster and hope nobody sees them
For 99% of people, there isn't a meaningful difference between these situations. I really don't think there are people out there that couldn't write a buffer overflow exploit in 5 years but could groom maintainers for trust in the meantime.
The code-writing and prose bit is the easy part. The social engineering and deception is hard enough for a human, and Turing-assured destruction for LLMs.
Comments
I don't think so. Sophisticated threat actors have been attacking the Open Source community for decades, adding AI into the equation changes nothing from my perspective. I'd actually imagine it's harder to get the precise and perfectly-obfuscated behavior you want while going through an LLM as a midpoint.
So my guess is no. Even in the case of the XZ backdoor, it was the incredible amount of social-engineering at play that enabled the hack. The developer had been conning the repo maintainer for years, which is easier achieved with human intellect than ChatGPT and a fake Github account.
Well it is true, the XZ hack was a major social-engineering feat. Regardless, we should not be underestimating the power of these technologies. If an LLM can respond to a request to generate code, what makes you think the attacker couldn't simply add things like "Insert a buffer overflow in the code..." or what have you. And especially as this technology progresses, the capabilities will no doubt dramatically increase. Having said that, I do hope you are right!
What makes you think an attacker couldn't do that without an LLM? I'm having a hard time understanding what changes in this scenario by introducing an AI.
Pretty much anyone can write a buffer overflow exploit with enough research. The harder part is getting your patches through code review, which is probably only hindered by having ChatGPT help you. Again... the past few decades have mostly gone off without a hitch, and there have been a lot of monkeys on typewriters hooked up to the internet.
I don't know, LLM's are very good at embedding subtle changes in their responses. They are also good at researching things like "best possible place to hide bugs", etc. Is it really a stretch to imagine that these technologies won't, if not now, but "someday soon" be used in such a way? I certainly don't think so.
Humans are really good at subtlety and research. I guess my point is this:
For 99% of people, there isn't a meaningful difference between these situations. I really don't think there are people out there that couldn't write a buffer overflow exploit in 5 years but could groom maintainers for trust in the meantime.The code-writing and prose bit is the easy part. The social engineering and deception is hard enough for a human, and Turing-assured destruction for LLMs.