That's kind of the entire premise of huggingface smolagent and while it does work really well when it works it also increase the challenges in rolling back failed actions
I guess one could in principle wrap the entire execution block into a distributed transaction, but llm try to make code that is robust, which works against this pattern as it makes hard to understand failure
Agree, the smolagent premise is good; but the hard part is handling execution, errors, etc.
For example, when the code execution fails mid-way, we really want the model to be able to pick up from where it failed (with the states of the variables at the time of failure) and be able to continue from there.
We've found that the LLM is able to generate correct code that picks up gracefully. The hard part now is building the runtime that makes that possible; we've something that works pretty well in many cases now in production at Lutra.
I think in principle you can make the entire API exposed to the llm idempotent so that it bicomes irrelevant for the backend wheter the llm replay the whole action or just the failed steps
That'd work well for read-only APIs, but we also want the LLMs to be able to update data, create documents, etc. Feels a bit harder when there are side-effects.
Comments
That's kind of the entire premise of huggingface smolagent and while it does work really well when it works it also increase the challenges in rolling back failed actions
I guess one could in principle wrap the entire execution block into a distributed transaction, but llm try to make code that is robust, which works against this pattern as it makes hard to understand failure
Agree, the smolagent premise is good; but the hard part is handling execution, errors, etc.
For example, when the code execution fails mid-way, we really want the model to be able to pick up from where it failed (with the states of the variables at the time of failure) and be able to continue from there.
We've found that the LLM is able to generate correct code that picks up gracefully. The hard part now is building the runtime that makes that possible; we've something that works pretty well in many cases now in production at Lutra.
I think in principle you can make the entire API exposed to the llm idempotent so that it bicomes irrelevant for the backend wheter the llm replay the whole action or just the failed steps
That'd work well for read-only APIs, but we also want the LLMs to be able to update data, create documents, etc. Feels a bit harder when there are side-effects.
Could you implement an actual state machine and have your agent work with that?
that's the langraph idea. each langraph node can then be a smolagent
latency tho, would be unbearable for real time.