The argument falls apart when the premise that you're chasing big O does.

Poor cache/memory/disk accesses can result in constant regressions so vast that a 'worse' algorithm actually fares better. Also, we tend to falsely settle on referring to 'O' rather than omega, theta, and average, even when we don't care about worse-case or contrived workloads. See quicksort and mergesort.

For a similar concept in another domain, see also the external memory model:

https://en.algorithmica.org/hpc/external-memory/model/