Comment on Robots.txt for the NYT has a specific exclusion for an 1996 news articleparentComments−remux5yThis was changed in 2019: https://www.robots-viewer.com/robots/checksum/64dc55e4da9cc8...robots.txt version from 2017: https://www.robots-viewer.com/robots/checksum/8029662cfb040c...−makomk5yProbably because that's around the time he seemingly really pissed off one or more former employees who went digging for more information on him: https://www.glassdoor.com/Reviews/Employee-Review-Romio-RVW2...
Comments
This was changed in 2019: https://www.robots-viewer.com/robots/checksum/64dc55e4da9cc8...
robots.txt version from 2017: https://www.robots-viewer.com/robots/checksum/8029662cfb040c...
Probably because that's around the time he seemingly really pissed off one or more former employees who went digging for more information on him: https://www.glassdoor.com/Reviews/Employee-Review-Romio-RVW2...