LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.
Set aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests.
Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.
Comments
I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea
LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.
Set aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests.
Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.
Because mutation testing (should) generate a lot of mutants, which you then need to run your unit tests against.
Yes but my point is it doesn't have to do that fast, as in unit test speeds.