Hacker News has been blocking Google for several days except for one crawler IP address. I think crawlers are especially hard on HN because they often request older pages that aren't in the cache.
This isn't new; see https://news.ycombinator.com/item?id=3277661 for example. I think HN is blocking most crawlers; Bing returns HN at #7 for a search for [hacker news] for example.
Comments
Hacker News has been blocking Google for several days except for one crawler IP address. I think crawlers are especially hard on HN because they often request older pages that aren't in the cache.
This isn't new; see https://news.ycombinator.com/item?id=3277661 for example. I think HN is blocking most crawlers; Bing returns HN at #7 for a search for [hacker news] for example.