Comment on Prompt caching for cheaper LLM tokensComments−duggan8moIt was a real facepalm moment when I realised we were busting the cache on every request by including date time near the top of the main prompt.Even just moving it to the bottom helped move a lot of our usage into cache.Probably went from something like 30-50% cached tokens to 50-70%.
Comments
It was a real facepalm moment when I realised we were busting the cache on every request by including date time near the top of the main prompt.
Even just moving it to the bottom helped move a lot of our usage into cache.
Probably went from something like 30-50% cached tokens to 50-70%.