Comment on Show HN: An open-source AI Gateway with integrated guardrailsComments−namanyayg2ysaw your tweet on X, nice work and congrats on launching!i'm curious about the caching mechanisms you've implemented to reduce repeated evaluations - are you using a traditional cache store like redis or something more bespoke?−roh26itOP2yWe use a bunch of caching mechanisms on the LLM requests themselves and extend the same to guardrails now.So there's 2 levels of cache - the LLM request itself might be cached (simple and semantic) and the guardrail response can be cached as well.We use a mix of a distributed kv store and a vector DB to actually store the data
Comments
saw your tweet on X, nice work and congrats on launching!
i'm curious about the caching mechanisms you've implemented to reduce repeated evaluations - are you using a traditional cache store like redis or something more bespoke?
We use a bunch of caching mechanisms on the LLM requests themselves and extend the same to guardrails now.
So there's 2 levels of cache - the LLM request itself might be cached (simple and semantic) and the guardrail response can be cached as well.
We use a mix of a distributed kv store and a vector DB to actually store the data