Thinking about this a bit more, the main issue is that the LLM is an attack vector. Hallucinations aren't a big deal if the action requires approval or can be easily undone. The only way right now I can see to make LLM interfaces 'safe' would be to give it an internal API and pass the user context so we can set up permissions that way. There are a very few cases where even that might not be enough (basically whenever you pass 'on-behalf-of' transactions _without_ direct approval from the origin via a temporary token - e.g. many backend payment APIs allow this).
However, if you have an internal API which is the real security boundary, you might as well expose it. The effort required to add an LLM is now an extra effort and that flips the incentives to where the LLM really adds value. So adding LLMs as interfaces really makes sense on Windows 11 (practically single user, hallucinations have limited cost) and GitHub (devs have local copy anyway), maybe on AWS (devs would like an easier interface, but there's a need to reassure about safety), and none on Stripe or MailGun (complex security scenarios, usually decided by finance/marketing department which don't care about integration difficulty).
Comments
Thinking about this a bit more, the main issue is that the LLM is an attack vector. Hallucinations aren't a big deal if the action requires approval or can be easily undone. The only way right now I can see to make LLM interfaces 'safe' would be to give it an internal API and pass the user context so we can set up permissions that way. There are a very few cases where even that might not be enough (basically whenever you pass 'on-behalf-of' transactions _without_ direct approval from the origin via a temporary token - e.g. many backend payment APIs allow this).
However, if you have an internal API which is the real security boundary, you might as well expose it. The effort required to add an LLM is now an extra effort and that flips the incentives to where the LLM really adds value. So adding LLMs as interfaces really makes sense on Windows 11 (practically single user, hallucinations have limited cost) and GitHub (devs have local copy anyway), maybe on AWS (devs would like an easier interface, but there's a need to reassure about safety), and none on Stripe or MailGun (complex security scenarios, usually decided by finance/marketing department which don't care about integration difficulty).