It's not that easy. Although some site owners enable protection without thinking about it, a lot of the time this is deliberate.

Site owners are increasingly concerned about AI crawlers taking their content without giving anything in return.

I'm afraid that this fight will only get more desperate as AI and AI agents start to replace more and more web browsing.