It's also basically impossible to block most AI crawlers. Most completely ignore robots.txt. Some claim to respect it but I've found the majority that claim to also ignore it. You have to block their useragent and then their IPs if they spoof it and use botnets like some China-based AI crawlers.