The setup is described in an earlier post linked in the second paragraph. Agents are given the zstd RFC and told to implement it, greenfield. Testing frameworks were used however the agents decided, which is to say, as mentioned repeatedly throughout the post, badly or not at all. Other factors that could conceivably affect the results can be assumed to be held constant, since the goal is to see what happens when you vary the instructions regarding which testing methodology should be used.
It's intended as a "test to see how effective agents are when guided by someone with no expertise in testing who's maybe heard that you should apply certain techniques or use certain libraries." What do you think would be a more effective test for that purpose?
So not "How well do agents use test/verification techniques?" like the title says then.
It just looks like a lot of effort to go to in a way that doesn't answer the question the title poses.
I'd love to see something that could guide how to set an agent up for success rather than this naive approach which only really answers the questions that most of the audience here probably don't have.
Comments
The setup is described in an earlier post linked in the second paragraph. Agents are given the zstd RFC and told to implement it, greenfield. Testing frameworks were used however the agents decided, which is to say, as mentioned repeatedly throughout the post, badly or not at all. Other factors that could conceivably affect the results can be assumed to be held constant, since the goal is to see what happens when you vary the instructions regarding which testing methodology should be used.
So basically "one-shot <complicated thing> using <x> testing methodology".
I don't think that's a very effective test of an agents ability.
It's intended as a "test to see how effective agents are when guided by someone with no expertise in testing who's maybe heard that you should apply certain techniques or use certain libraries." What do you think would be a more effective test for that purpose?
So not "How well do agents use test/verification techniques?" like the title says then.
It just looks like a lot of effort to go to in a way that doesn't answer the question the title poses.
I'd love to see something that could guide how to set an agent up for success rather than this naive approach which only really answers the questions that most of the audience here probably don't have.