Comment on Benchmarking coding agents on Databricks' multi-million line codebaseComments−lukax2moCould it be that users of Pi are more senior and know better how to prompt and that's why the pass rate is higher?−rootatixww32mothey explain this is a benchmark, all models/harnesses receive the same prompt
Comments
Could it be that users of Pi are more senior and know better how to prompt and that's why the pass rate is higher?
they explain this is a benchmark, all models/harnesses receive the same prompt