Agreed. The author makes the point that middleware is trying to solve valid problems. I think the issue is that there are better ways to address it by “avoiding the middleware all together”. Especially since it introduces other 3rd parties into the mix
Yea, half of this seems like problems we've already solved for APIs generally, the other half is LLM specific, like prompt management, log/mon, response quality in prod, real time feedback
It seems like tools from our current ops ought to work just fine for the non-LLM uniqueness. At the same time, Datadog is pretty popular for a managed experience, and the LLM services as proxies kind of fit that model
This project looks great and aligns with my thinking
Comments
Agreed. The author makes the point that middleware is trying to solve valid problems. I think the issue is that there are better ways to address it by “avoiding the middleware all together”. Especially since it introduces other 3rd parties into the mix
Yea, half of this seems like problems we've already solved for APIs generally, the other half is LLM specific, like prompt management, log/mon, response quality in prod, real time feedback
It seems like tools from our current ops ought to work just fine for the non-LLM uniqueness. At the same time, Datadog is pretty popular for a managed experience, and the LLM services as proxies kind of fit that model
This project looks great and aligns with my thinking