This has been pretty well studied. Pangram is about as good at an expert human with a 98% detection rate and <2% false positive, for flagging AI generated text.
I'm open to being convinced, but the study you've linked me is limited specifically to professionally written, proofread journalism from 8 publications over a span of a little over a single year.
It's not exactly a wide breadth of varied text for such an all-encompassing task.
The primary reason I am not convinced is because it (and several other tools) frequently classify my own writing as AI-generated. Especially my technical writeups.
Now, I could just assume I'm an outlier within that small false-positive rate. But I find it far more likely that the efficacy of these tools is much lower than advertised and they rely on confirmation bias.
Comments
I’m not saying whether I think you’re right or wrong, but AI text detectors don’t work and never will.
This has been pretty well studied. Pangram is about as good at an expert human with a 98% detection rate and <2% false positive, for flagging AI generated text.
https://arxiv.org/pdf/2501.15654
I'm open to being convinced, but the study you've linked me is limited specifically to professionally written, proofread journalism from 8 publications over a span of a little over a single year.
It's not exactly a wide breadth of varied text for such an all-encompassing task.
https://www.pangram.com/blog/third-party-pangram-evals
You can also just try it yourself I guess really what convinced me was how it perfectly agrees with my own judgement.
The primary reason I am not convinced is because it (and several other tools) frequently classify my own writing as AI-generated. Especially my technical writeups.
Now, I could just assume I'm an outlier within that small false-positive rate. But I find it far more likely that the efficacy of these tools is much lower than advertised and they rely on confirmation bias.
Doesn't stop charlatans from creating startups promising to make it work :)