- litellm listens on 127.0.0.1:4242 with some pre-shared key, give the credentials to the harness.
- Tools can exfiltrate any data accessible to them given network access, so all tools run inside a VM or a container targeted to the specific tasks. For example, I've made podman spawner that runs a 'tool server' inside a container, and all of my custom harness' tool requests go to that container (which can be barred from the network). The workspace for that LLM is solely inside the ephemeral container, but it can get bind mounts from outside if needed.
Comments
- litellm listens on 127.0.0.1:4242 with some pre-shared key, give the credentials to the harness. - Tools can exfiltrate any data accessible to them given network access, so all tools run inside a VM or a container targeted to the specific tasks. For example, I've made podman spawner that runs a 'tool server' inside a container, and all of my custom harness' tool requests go to that container (which can be barred from the network). The workspace for that LLM is solely inside the ephemeral container, but it can get bind mounts from outside if needed.