Though I ran the numbers, and even with a 0.02% false positive rate, that works out to about 6000 students falsely accused every semester, per university.
It's also confirmed in third-party studies like this https://arxiv.org/html/2501.15654v1 (which also confirms that humans that have experience with LLMs are very good at spotting them).
> Detect AI-generated content with 99.98% accuracy.
Though I ran the numbers, and even with a 0.02% false positive rate
They don't say that the false positive is 0.02%, only that the accuracy is 0.02%. All we know for certain is that the false positive and false negatives added together result in 0.02%.
Comments
Didn't Pangram claim to have a >99% success rate?
https://news.ycombinator.com/item?id=48667761
From their homepage:
Are they wrong?
Though I ran the numbers, and even with a 0.02% false positive rate, that works out to about 6000 students falsely accused every semester, per university.
It's also confirmed in third-party studies like this https://arxiv.org/html/2501.15654v1 (which also confirms that humans that have experience with LLMs are very good at spotting them).
They don't say that the false positive is 0.02%, only that the accuracy is 0.02%. All we know for certain is that the false positive and false negatives added together result in 0.02%.
Yeah, my bad. Well they don't exactly guard against that interpretation. "Up to 6000 falsely accused per semester!" doesn't exactly ease the mind!