do you read the code and test code? how do you know that the tests are any good? in my experience a lot of the time it will just mock the expected outcome and then it passes but it doesn't actually test behaviour.
Most of the time it works as you said, but I implemented one rule that fixed these issues for me: my agent doesn't count a test until it has watched it fail.
The process works the following way: the agent writes the test, then it deletes the code it covers, reruns the test, confirms it goes red, and finally puts the code back.
Here is what made me start doing this. An e2e test for a touch gesture was green for weeks. I deleted the gesture handler, and the test stayed green, because Playwright's synthetic touch isn't actually a touch, so the test had never once exercised the thing it was named after. Reading it would never have shown me that.
It's discipline, not tooling. I don't run mutation testing; it's manual and only around whatever I just changed. And to be straight about your first question: it's the agent reading that test code, not me. Which is exactly why I don't rely on reading to catch this. Breaking the code and watching the test go red doesn't care who read it.
I use a combination of automated and manual testing. I spent a bunch of time setting up playwright testing at the start of my project, and then I shunt that through ZAP as well so that it can have a good understanding of how things work.
If gives me a pretty solid map of what is working well, and aids in driving through security testing.
Still not a replacement for manual testing. Us meatbags still have a knack for royally messing up the expected use cases :P
Comments
do you read the code and test code? how do you know that the tests are any good? in my experience a lot of the time it will just mock the expected outcome and then it passes but it doesn't actually test behaviour.
Most of the time it works as you said, but I implemented one rule that fixed these issues for me: my agent doesn't count a test until it has watched it fail.
The process works the following way: the agent writes the test, then it deletes the code it covers, reruns the test, confirms it goes red, and finally puts the code back.
Here is what made me start doing this. An e2e test for a touch gesture was green for weeks. I deleted the gesture handler, and the test stayed green, because Playwright's synthetic touch isn't actually a touch, so the test had never once exercised the thing it was named after. Reading it would never have shown me that.
It's discipline, not tooling. I don't run mutation testing; it's manual and only around whatever I just changed. And to be straight about your first question: it's the agent reading that test code, not me. Which is exactly why I don't rely on reading to catch this. Breaking the code and watching the test go red doesn't care who read it.
I use a combination of automated and manual testing. I spent a bunch of time setting up playwright testing at the start of my project, and then I shunt that through ZAP as well so that it can have a good understanding of how things work.
If gives me a pretty solid map of what is working well, and aids in driving through security testing.
Still not a replacement for manual testing. Us meatbags still have a knack for royally messing up the expected use cases :P
Sadly I can't rely on manual testing in my professional work although for personal projects it's how I do it too