Crawlers visiting every page on your website is not the main problem with the unauthenticated web.

The amount of spam that happens when you let people freely post is a much bigger problem.