Addendum: to answer the obvious question, it's very difficult to test on unseen "OOD" data, because you don't have the "ground truth" for it. It is possible to do such "extrinsic" evaluation of a machine learning system but it takes time - time generally spent with the system operating in a real-world environment ("in deployment"). By that time, the system you want to test might already be obsolete, because better (lab-based) results have been published and there's no point in trying to publish an extrinsic evaluation of an old system, especially if the extrinsic evaluation shows that performance degraded by 20% compared to the lab-based evaluation, let alone the new SOTA.

So results on truly-unseen, OOD data are never published, if they are ever collected at all, and the field advances one meaningless published benchmark-beating SOTA result at a time. In the end, despite all the hype, nobody has any idea how well neural networks, er, work, in practice. We only know how well they do on test datasets.

Oh, alright. Some people who deploy large neural nets for google and friends have an intuition about how well they work. A ... feeling.