Good point, but I think Crichton’s original example still makes some sense. One’s choice of newspaper could affect the trustworthiness of articles by multiple journalists:
• Some articles may be edited by the same editor. If you see inaccuracies in an article by one journalist, you may worry that the editor is bad at fact-checking.
• Even if every article is edited by a different editor, if the newspaper’s work environment is rushed such that editors are told not to spend more than two minutes thinking about accuracy, it might have more inaccuracies than a newspaper that allows more fact-checking time. (In other words, newspapers have their own publication standards.)
• Newspapers set standards for what journalists to hire and retain. A newspaper that keeps paying for articles by a journalist who have been known to make mistakes is more likely to not care about mistakes they know of by their other journalists.
Bringing this analogy back to LLMs: if you see one LLM make mistakes, you should be suspicious of the accuracy of a competing LLM to the extent that the competing LLM has similar “journalism” and “publication standards”.
Comments
Good point, but I think Crichton’s original example still makes some sense. One’s choice of newspaper could affect the trustworthiness of articles by multiple journalists:
• Some articles may be edited by the same editor. If you see inaccuracies in an article by one journalist, you may worry that the editor is bad at fact-checking.
• Even if every article is edited by a different editor, if the newspaper’s work environment is rushed such that editors are told not to spend more than two minutes thinking about accuracy, it might have more inaccuracies than a newspaper that allows more fact-checking time. (In other words, newspapers have their own publication standards.)
• Newspapers set standards for what journalists to hire and retain. A newspaper that keeps paying for articles by a journalist who have been known to make mistakes is more likely to not care about mistakes they know of by their other journalists.
Bringing this analogy back to LLMs: if you see one LLM make mistakes, you should be suspicious of the accuracy of a competing LLM to the extent that the competing LLM has similar “journalism” and “publication standards”.