Skip to content

Comment on Messing with scraper botsparent

Comments

Modern browsers support brotli or zstd, which is a lot better in terms of compression. Perhaps not as good for on-the-fly compression, but static assets can get a nice compression benefit out of it.

With toxic AI scrapers like Perplexity moving more and more to headless web browsers to bypass bot blocks, I think a brotli bomb (100GB of \0 can be compressed to about 78KiB with Brotli) would be quite effective.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.