That would depend on what LLM are you using and the average tokens per answer. For example the average cost of the user query in my system is about 0.07 if you add 0.01 you are adding 15% per query (+-). Then, take into consideration how much value does RAG add to the answer Vs using other method.
Comments
Is $0.01 per retrieval a lot, or not?
That would depend on what LLM are you using and the average tokens per answer. For example the average cost of the user query in my system is about 0.07 if you add 0.01 you are adding 15% per query (+-). Then, take into consideration how much value does RAG add to the answer Vs using other method.
Edit: typo in costs
Interesting. Thanks for sharing.