Before LLMs we had a pretty good idea of security boundaries in software. Applications didn’t trust user input. Operating systems didn’t trust applications. Services and processes didn’t trust each other. There were always tokens, scopes, delegated grants.
Suddenly every AI company’s security model seems to be to say “pretty please” to a non-deterministic machine and hope for the best. And if there is a security failure instead of accepting blame they go “well we can’t help it, our model is too intelligent”.
Why can’t we give agents a shell with permissions for programs and file system access controlled by Unix permissions?
This seemed to be a solved problem back in the systems where many users were logged into one machine and the admins had to keep everyone from impacting each other.
In a recent full discloure I was reading about a CVE of an LLM agent, the vendor installed a “secure sandbox VM” and then just shared a host’s filesystem read/write to the agent’s VM.
Two problems. First, it is remarkably difficult to come up with a set of programs that it is safe to let the restricted user use. Second, it is remarkably difficult to make the restricted environment useful enough if you're really serious about allowing only safe programs to be used. Try to make it useful enough and you end up with escapes everywhere.
Sure but then the file protections also apply to the programs used by the agent. If you can strictly limit the agent to only modifying files in the source code directory of a single repository, that greatly limits the blast radius of damage.
Suddenly determinism has other meanings than "same input leads to the same output". I'm still confused by this and not sure how it happened so easily, but it seems to be accepted by everyone now.
Philosophical determinism is pretty similar in spirit to physics/CS (roughly same cause = same effect, no free will/side effects), and what parent calls determinism seems to be neither. It's rather about natural language prompts not being formally specified in the first place. Unreliability inherent to any intelligence (artificial or natural)? but not determinism I think. The word somehow got universally hijacked.
Ah yes. So can we just lump most LLM/AI problems into just "people" problems and stop falling for the popular mainstream straw man of "Look at the tech".
Comments
Before LLMs we had a pretty good idea of security boundaries in software. Applications didn’t trust user input. Operating systems didn’t trust applications. Services and processes didn’t trust each other. There were always tokens, scopes, delegated grants.
Suddenly every AI company’s security model seems to be to say “pretty please” to a non-deterministic machine and hope for the best. And if there is a security failure instead of accepting blame they go “well we can’t help it, our model is too intelligent”.
Amen.
Why can’t we give agents a shell with permissions for programs and file system access controlled by Unix permissions?
This seemed to be a solved problem back in the systems where many users were logged into one machine and the admins had to keep everyone from impacting each other.
It would loose automagic?
In a recent full discloure I was reading about a CVE of an LLM agent, the vendor installed a “secure sandbox VM” and then just shared a host’s filesystem read/write to the agent’s VM.
We never solved the restricted shell problem for humans.
Can you elaborate?
Read, write, execute privileges on files and directories goes a long way. What's missing?
The biggest is Internet access, or networking in general, I suppose.
Two problems. First, it is remarkably difficult to come up with a set of programs that it is safe to let the restricted user use. Second, it is remarkably difficult to make the restricted environment useful enough if you're really serious about allowing only safe programs to be used. Try to make it useful enough and you end up with escapes everywhere.
Sure but then the file protections also apply to the programs used by the agent. If you can strictly limit the agent to only modifying files in the source code directory of a single repository, that greatly limits the blast radius of damage.
But that's not useful. That's the problem.
Suddenly determinism has other meanings than "same input leads to the same output". I'm still confused by this and not sure how it happened so easily, but it seems to be accepted by everyone now.
I may say you're thinking of the term "deterministic algorithm" as determinism is more of a philosophical definition that has evolved over the years.
Philosophical determinism is pretty similar in spirit to physics/CS (roughly same cause = same effect, no free will/side effects), and what parent calls determinism seems to be neither. It's rather about natural language prompts not being formally specified in the first place. Unreliability inherent to any intelligence (artificial or natural)? but not determinism I think. The word somehow got universally hijacked.
And the "oh noes! our model is too intelligent!!" thing is advertising.
Before LLMs we didnt have much accountability from leadership, that was eroding over time. After LLMs we still dont.
People problem, not tech problem. Can't solve people problems with tech, you can only make them worse, and wider.
Ah yes. So can we just lump most LLM/AI problems into just "people" problems and stop falling for the popular mainstream straw man of "Look at the tech".