that can be achieved by storing user traces and then running evals on own model vs competitor model. you don't need to route live customer requests to a competitor for this.
For agentic tasks where the model outputs tool calls that run on the customer's computer, you can't just store and eval later, because then the execution environment is no longer available.
Comments
that can be achieved by storing user traces and then running evals on own model vs competitor model. you don't need to route live customer requests to a competitor for this.
For agentic tasks where the model outputs tool calls that run on the customer's computer, you can't just store and eval later, because then the execution environment is no longer available.