Comment on Benchmarking coding agents on Databricks' multi-million line codebaseparentComments−rootatixww32mofor that you would need to compare the same task implemented in two different languages - C# and Python for example, no?−yodon1moThe advantage of their dataset is it's large enough that you have a statistical number of PR's, allowing you to treat the average cost of Python PR's vs the average cost of C# costs (or etc.) as statistically meaningful.
Comments
for that you would need to compare the same task implemented in two different languages - C# and Python for example, no?
The advantage of their dataset is it's large enough that you have a statistical number of PR's, allowing you to treat the average cost of Python PR's vs the average cost of C# costs (or etc.) as statistically meaningful.