You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
Was your prompt, "make lazy ineffectual tests?" Tests are code as well, they take the same effort to create, if not more. You need to specify the properties you want to test.
I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.
Isn't that just a form of Dunning-Kruger effect related to the parts that person is bad in? It looks good because they don't have the depth of knowledge/expertise to see the deficiencies.
Comments
You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
I thought so too but it's probably because I never review the tests. I made an exception recently and found tons of
or tests for keywords in prompts:Was your prompt, "make lazy ineffectual tests?" Tests are code as well, they take the same effort to create, if not more. You need to specify the properties you want to test.
Most developers are not good at testing. That's why the role of test engineer exists.
I've seen in multiple projects things like assertTrue(true).
I'm sure the agent is better in testing than average enterprise developer.
big tech is moving / has moved away from the role.
Unfortunately for the users, testing has been pushed to them. You can deploy whatever slop you want - nobody cares. More lines of code!
I don't think these results show that.
I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.
Isn't that just a form of Dunning-Kruger effect related to the parts that person is bad in? It looks good because they don't have the depth of knowledge/expertise to see the deficiencies.