Rust/Python hybrids are also quite well established, so it allows us to work off of a stable base - while ensuring we can deliver
- low memory utilization
- low request latency overhead
which is the primary goal here (be fast, lightweight and cheap to deploy).
2. LiteLLM has the most mature, and broadest range of unified api's x providers. This means you do not need to give developers raw LLM API keys, ever.
We see devs using/building agents that consume a lot of different API's - responses api, realtime, chat completions, messages - and no matter what they use we want them to be able to switch across providers without if/else statements in their code.
I can't comment on others, but that's our goal and what we work on doing everyday. So I would trust that we do it well.
Beyond that, we're also growing to become the single point of access for all AI resources. This makes it a lot easier when building agents, because you can give an agent 1 key, and it will have access to LLM's + MCP's (and in the future other resources like skills, api credentials, sandbox api's, etc.).
Comments
1. I think rust is the better long term solution over go. Especially for a gateway. You can see other proxy's (e.g. cloudflare's pingora - https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...) doing the same thing.
Rust/Python hybrids are also quite well established, so it allows us to work off of a stable base - while ensuring we can deliver - low memory utilization - low request latency overhead
which is the primary goal here (be fast, lightweight and cheap to deploy).
2. LiteLLM has the most mature, and broadest range of unified api's x providers. This means you do not need to give developers raw LLM API keys, ever.
We see devs using/building agents that consume a lot of different API's - responses api, realtime, chat completions, messages - and no matter what they use we want them to be able to switch across providers without if/else statements in their code.
I can't comment on others, but that's our goal and what we work on doing everyday. So I would trust that we do it well.
Beyond that, we're also growing to become the single point of access for all AI resources. This makes it a lot easier when building agents, because you can give an agent 1 key, and it will have access to LLM's + MCP's (and in the future other resources like skills, api credentials, sandbox api's, etc.).