Skip to content

Comment on Show HN: Crawlee – Web scraping and browser automation library for Node.jsparent

Comments

In crawlee, you can use the useIncognitoPages option to create a separate context for each page https://crawlee.dev/api/browser-pool/class/LaunchContext#use... Not sure if it will be enough to offset your RAM requirements.

From our experience, RAM is not the limiting factor. It's the CPU. You need at least 1 CPU core for the modern browsers to work reliably at scale so if you're using a container that has 1GB ram and 0.25 core, it's just not worth it. If you have access to containers that have strong CPUs and not a lot of RAM, then it's a different story.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.