I think this is great, and the next frontier is to analyze how well calls can be handled by a local model. Realistically, to do that usefully requires response time as a new dimension of judging: Can a local model provide an acceptably accurate response in an acceptable amount of time?
Thanks. Yes, local models are gaining a lot of traction.
The measure/judge step uses LiteLLM, so it does sample local models. I just tested "--candidates ollama/llama3.2:1b", and that works - ignoring the lack of rich UX for local/unpriced models, as I was focussing on cloud cost, but you've inspired me to give this area some polish.
Noted: "response time as a new dimension of judging" - Added to the roadmap.
If you experience any issues with LiteLLM, you may try GoModel - the AI Gateway I'm working on. It consumes ~60x less resources and is more reliable :)
Comments
I think this is great, and the next frontier is to analyze how well calls can be handled by a local model. Realistically, to do that usefully requires response time as a new dimension of judging: Can a local model provide an acceptably accurate response in an acceptable amount of time?
Thanks. Yes, local models are gaining a lot of traction.
The measure/judge step uses LiteLLM, so it does sample local models. I just tested "--candidates ollama/llama3.2:1b", and that works - ignoring the lack of rich UX for local/unpriced models, as I was focussing on cloud cost, but you've inspired me to give this area some polish.
Noted: "response time as a new dimension of judging" - Added to the roadmap.
Try it and let me know if you hit a wall.
If you experience any issues with LiteLLM, you may try GoModel - the AI Gateway I'm working on. It consumes ~60x less resources and is more reliable :)