Show HN: Flawd is mutation testing for the AI era

Hey, all! Flawd is a mutation testing tool that can target five languages (Python, JS, TS, Go, Rust) and it runs locally as a single binary on your machine. Importantly (for many), your code never leaves your machine and all processing takes place directly on your dev box or CI runner.

Mutation testing is a process in which faults (known as mutants) are intentionally injected into your codebase, tests are run and any faults which are not detected by your tests are known as survivors (or surviving mutants). Survivors can often indicate that your test assertions are not focused on expected behavior or might be missing altogether. Whereas code coverage can tell you which code was exercised during a test run, mutation testing can provide some insight regarding the strength of the assertions and whether or not certain classes of bugs would be caught.

With increasingly more code and tests being authored by coding agents, it seemed like a good time to build a tool like this. Mutation testing is not new, but many of the tools are either single-language specific or haven’t been designed with an agentic workflow in mind, so those are itches we wanted to scratch. Models continue to become more powerful and effective, but at some point there is still a “who watches the watchmen” scenario and we hope Flawd can provide one check towards that balance.

We recently ran Flawd against 10 established open source projects and you can find the full writeup here: https://fixture.dev/writing/we-mutation-tested-10-projects

Also, you can see what a sample report looks like here: https://fixture.dev/flawd/sample-report . Flawd can also generate machine readable reports that can be handed off to coding agents for test strengthening tasks.

Feel free to ask my anything about Flawd and thanks for having a look!

alxwrd 11d

Would love to try this, but it appears the trial requests are broken

    Content-Security-Policy: The page’s settings blocked a script (script-src-elem) at https://challenges.cloudflare.com/turnstile/v0/api.js?render=explicit from being executed because it violates the following directive: “script-src 'self' 'sha256-SMalEXjN5ltC0WTMOtJTO7adqjVmsBbKVQMxVs4wkNM='” app.zp5tf3yd.js:1123:459


    >>> POST https://www.fixture.dev/api/eval/request

    <<< {"error":"turnstile_token is required","code":"VALIDATION_ERROR"}

foharaOP11d

Ouch, thanks for the heads up! Will have a fix up shortly.

foharaOP10d

Fixed. Thanks again for the report and sorry for the hassle!

devo_shn 11d

Okay, this looks great! I have been struggling with mutation testing on a personal project built in Python. Just a few things: how does it generate mutants? Does it need specific testing libraries (like pytest) and then create mutants out of those tests? For the particular project, I created a small testing library as I felt pytest would be overkill, curious if it'll be able to help me too.

foharaOP11d

Thanks! It generates mutant by parsing the source code (via tree-sitter) and then using a collection of operators to make changes such as invert booleans, flip comparison operators, etc.

It is built with certain testing libraries in mind, especially for defaults, but it is also extensible such that you should be able to get it working with your custom testing library. The `flawd init` command should provide some meaningful context, and I would be happy to lend a hand with config if you're up for sharing some more context. Feel free to contact me through the site or via email (in profile)

Comments