Amazing work! But deeply frustrating on the lack of reproducibility and how far off this is from a proper SDLC.
How can I inspect exactly how agents were prompted? How can I reproduce this setup and try out my own AI setup? Am I missing a link to a repo somewhere?
These evals probably match how many people are using AI, but its not the full package of how we know software development needs to be done, and not how I do it with AI. The closest would be "Audit" which scored the highest when using xhigh- that actually incorporates a review cycle- something we know is the most important part of the software development process for code correctness (design/specification is not as much of an issue in this problem since the task is to write code against an existing spec). However, we don't know what instructions they have in their "Audit".
I would love to benchmark my own flow [1] if I can be given their exact problem. What it does is (assuming there is already a solid spec)
* plan with expensive model. Review the plan.
* implement with cheap model. Review for spec compliance and code quality.
* Reviews are done adversarially from the expensive model with a fresh context.
* ensure that verifications (automated or manual) are performed.
For non-trivial changes, the review and verification process almost always catch significant issues.
The workflow does use TDD. I do find useless tests being written and I need to dig into this part of the workflow a lot more, so its great to see that aspect of this article. My experience writing software has taught me that code must be written to be easy to test, but not necessarily done TDD style.
As someone who also spent a bunch of time benchmarking various techniques, this is as good as you can do without publishing a formal versioned benchmark suite that you want to maintain and run forever at immense cost to yourself. If you actually go to benchmark your own flow as you mentioned, you will quickly run into like a dozen problems that discourage you from publishing.
- Are you sure that temperature and other nondeterminism isn't affecting your output?
- Are you sure you're not being routed through an A/B test at this moment?
- Are you sure there's not a bug affecting the model at this moment?
- Are you sure that you picked the right model and effort level?
- Are you sure that your result generalizes across providers?
- Are you sure that you set up the correct level of sandboxing and the agent can't e.g. look at a sister directory or git history in the current directory for answers?
- Are you sure that the agent isn't leaking answers in memory or its conversation history?
- Are you sure that tool calls aren't somehow affecting results?
- Are you sure that your results are robust, i.e. you see the same results with mild tweaks to the prompt?
- Are you comfortable keeping your blog post live when your results are invalidated next week with the next model launch?
And that's just a quick list off the top of my head.
I personally decided that it wasn't worth it, I'm glad that Dan decided to publish his. Frankly I think we could use a lot more of these "I ran these 2 techniques side by side and here's what I saw" anecdata, because most people who promote prompt techniques can't produce a single prompt they ran twice because they never actually tested it per se.
I am really grateful that he’s publishing as well.
I am just asking for the bare minimum to actually understand what has been tested and for reproducibility of methods- the publication of the prompts. It would take a lot less time than all the guess work analysis write up and be a lot more useful.
Without the prompts the rest of your questions about reproducibility are moot.
This post boils down to, "I tried a bunch of things that didn't work very well". But we don't actually know what he tried. Everything hinges on Dan's prompting skills.
Comments
Amazing work! But deeply frustrating on the lack of reproducibility and how far off this is from a proper SDLC.
How can I inspect exactly how agents were prompted? How can I reproduce this setup and try out my own AI setup? Am I missing a link to a repo somewhere?
These evals probably match how many people are using AI, but its not the full package of how we know software development needs to be done, and not how I do it with AI. The closest would be "Audit" which scored the highest when using xhigh- that actually incorporates a review cycle- something we know is the most important part of the software development process for code correctness (design/specification is not as much of an issue in this problem since the task is to write code against an existing spec). However, we don't know what instructions they have in their "Audit".
I would love to benchmark my own flow [1] if I can be given their exact problem. What it does is (assuming there is already a solid spec)
For non-trivial changes, the review and verification process almost always catch significant issues.The workflow does use TDD. I do find useless tests being written and I need to dig into this part of the workflow a lot more, so its great to see that aspect of this article. My experience writing software has taught me that code must be written to be easy to test, but not necessarily done TDD style.
[1] https://github.com/gregwebs/skills-sdlc/
As someone who also spent a bunch of time benchmarking various techniques, this is as good as you can do without publishing a formal versioned benchmark suite that you want to maintain and run forever at immense cost to yourself. If you actually go to benchmark your own flow as you mentioned, you will quickly run into like a dozen problems that discourage you from publishing.
- Are you sure that temperature and other nondeterminism isn't affecting your output?
- Are you sure you're not being routed through an A/B test at this moment?
- Are you sure there's not a bug affecting the model at this moment?
- Are you sure that you picked the right model and effort level?
- Are you sure that your result generalizes across providers?
- Are you sure that you set up the correct level of sandboxing and the agent can't e.g. look at a sister directory or git history in the current directory for answers?
- Are you sure that the agent isn't leaking answers in memory or its conversation history?
- Are you sure that tool calls aren't somehow affecting results?
- Are you sure that your results are robust, i.e. you see the same results with mild tweaks to the prompt?
- Are you comfortable keeping your blog post live when your results are invalidated next week with the next model launch?
And that's just a quick list off the top of my head.
I personally decided that it wasn't worth it, I'm glad that Dan decided to publish his. Frankly I think we could use a lot more of these "I ran these 2 techniques side by side and here's what I saw" anecdata, because most people who promote prompt techniques can't produce a single prompt they ran twice because they never actually tested it per se.
Edited to add: formatting + the word "promote"
I am really grateful that he’s publishing as well.
I am just asking for the bare minimum to actually understand what has been tested and for reproducibility of methods- the publication of the prompts. It would take a lot less time than all the guess work analysis write up and be a lot more useful.
Without the prompts the rest of your questions about reproducibility are moot.
This post boils down to, "I tried a bunch of things that didn't work very well". But we don't actually know what he tried. Everything hinges on Dan's prompting skills.