Skip to content

Comment on Python Headless Web Browser Scraping on Amazon Linuxparent

Comments

To follow up to your edit, that may be true in one case. But it's perfectly reasonable to navigate via clicking, anything in the navigate API, JS actions, meta refreshes, and so on. Even in that one case, most people would expect redirects to be followed and basic auth protected pages to submit. Again, all tractable problems, but ones that are likely better handled by an interstitial layer where you can see the entire chain of requests & responses.

Browsers do see the entire chain of requests and responses. All of it. Some browsers make that information available externally. I just don't see why a browser remote control solution like Selenium shouldn't pass on as much of this information as possible.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.