Skip to content

Comment on Scanning for Pangram Errors

Comments

Note that any good binary classifier (like "this is AI writing" for Pangram) should maximize the following four probabilities:

1. P(actually true | predicted true) (= true positive rate, recall, sensitivity)

2. P(actually false | predicted false) (= true negative rate, specificity)

3. P(predicted true | actually true) (= positive predictive value, precision)

4. P(predicted false | actually false) (= negative predictive value)

If you just try to minimize the share of false positives, you are only maximizing 1.

The measure which maximizes all four values is the binary Pearson correlation, also called "phi coefficient" or "MCC". It is 1 if the classifier is always correct, 0 if it guesses randomly (its predictions are statistically independent of the facts) and -1 if it always predicts the opposite of the true value.

They report false negative rate (FNR) here:

https://www.pangram.com/research/model-card/pangram-3-3

  Academic Writing  FNR=0.00%  N=48443
  Creative Writing  FNR=0.23%  N=41940
  RAID              FNR=0.93%  N=66855
Recall = 1 - FNR

Yeah, that corresponds to a high value of my 1. The other three values are also important though.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.