Skip to content

Comment on Ask HN: Best web scraping toolset for 2024?parent

Comments

I've done this sort of thing and it does work well. The caveat is you usually need a cookie/token after logging in, so you still need to fake that part as well as manage the session state.

It does work well, though some sites certainly detect the behavior after a while.

Right..

I'm in an antiquated space (logistics and wholesale) so most websites aren't hard to scrape and even mimic the auth.

The other avenue I've had success with is sniffing android app traffic to reverse engineer the API calls.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.