Comment on Can LLMs Do Accounting?Comments−yunyuOP1yLLMs are on the verge of replacing data scientists and investment bankers. But can they perform simple accounting tasks for a real business?We built AccountingBench, a test where LLMs must "close the books" for a real SaaS business using 1 year of Stripe/Ramp/Rippling/Mercury data.Claude 4 and Grok 4 start strong - within 1% of human CPA baselines in month 1.But as time progresses, all models inevitably accumulate compounding errors and exhibit erratic behavior, causing significant deviations.That said, the early accuracy here is promising. With targeted post-training, models may be able to replace humans for this kind of work.−simmerup1yAccounting isn't really the type of thing that can accept errors though is it?Like it needs to be 0% error rate−yunyuOP1yA certain level of errors is tolerable/inevitable. But the accountants need to be able to correct for them once they build up
Comments
LLMs are on the verge of replacing data scientists and investment bankers. But can they perform simple accounting tasks for a real business?
We built AccountingBench, a test where LLMs must "close the books" for a real SaaS business using 1 year of Stripe/Ramp/Rippling/Mercury data.
Claude 4 and Grok 4 start strong - within 1% of human CPA baselines in month 1.
But as time progresses, all models inevitably accumulate compounding errors and exhibit erratic behavior, causing significant deviations.
That said, the early accuracy here is promising. With targeted post-training, models may be able to replace humans for this kind of work.
Accounting isn't really the type of thing that can accept errors though is it?
Like it needs to be 0% error rate
A certain level of errors is tolerable/inevitable. But the accountants need to be able to correct for them once they build up