I don't know, LLM's are very good at embedding subtle changes in their responses. They are also good at researching things like "best possible place to hide bugs", etc. Is it really a stretch to imagine that these technologies won't, if not now, but "someday soon" be used in such a way? I certainly don't think so.
Humans are really good at subtlety and research. I guess my point is this:
pre-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge malicious patches and hope nobody sees them
post-LLM era XZ-style exploit: takes 5 years to attain developer trust, then you merge patches that were written slightly faster and hope nobody sees them
For 99% of people, there isn't a meaningful difference between these situations. I really don't think there are people out there that couldn't write a buffer overflow exploit in 5 years but could groom maintainers for trust in the meantime.
The code-writing and prose bit is the easy part. The social engineering and deception is hard enough for a human, and Turing-assured destruction for LLMs.
Comments
I don't know, LLM's are very good at embedding subtle changes in their responses. They are also good at researching things like "best possible place to hide bugs", etc. Is it really a stretch to imagine that these technologies won't, if not now, but "someday soon" be used in such a way? I certainly don't think so.
Humans are really good at subtlety and research. I guess my point is this:
For 99% of people, there isn't a meaningful difference between these situations. I really don't think there are people out there that couldn't write a buffer overflow exploit in 5 years but could groom maintainers for trust in the meantime.The code-writing and prose bit is the easy part. The social engineering and deception is hard enough for a human, and Turing-assured destruction for LLMs.