This looks really great. However, I can't find examples of how to handle scraping behind a login or a paywall, without having to 'type' credentials every time.
You can use a headless browser (would recommend PlaywrightCrawler) to log in once and then use the session cookie until it expires in any crawler. When it expires, you can re-login and repeat the process.
Comments
This looks really great. However, I can't find examples of how to handle scraping behind a login or a paywall, without having to 'type' credentials every time.
Just found this on the Apify documentation: https://docs.apify.com/tutorials/log-into-a-website-using-pu...
Is there a similar guide for Crawlee?
The example uses Crawlee already, you can just remove the
import { Actor } from 'apify';
and then all references to Actor and either remove them or replace them with Crawlee functions.
E.g. await Actor.openKeyValueStore() should be replaced with KeyValueStore.open()
It makes sense to add a separate example for Crawlee though. But it's true that it does not exist yet.
You can use a headless browser (would recommend PlaywrightCrawler) to log in once and then use the session cookie until it expires in any crawler. When it expires, you can re-login and repeat the process.