It’s not widespread because it’s much more complicated than making an http request and reading the results from the body. You don’t spin up a browser, much less the full GUI, unless it’s a last resort.
Well, very much yes and no to that claim. Sure, for someone who’s comfortable in the shell, the first step is lighter with curl or wget. But the next step – parsing — is a lot less obvious. And it’s all much more likely to fail on websites that assume a (logged-in?) user in a browser, accepting cookies, executing JS, and so on.
Comments
Yes: I find it surprising that this isn't a more widespread approach. It's how I've taught web scraping to my PhD students for some years.
https://github.com/jawj/web-scraping-for-researchers
It’s not widespread because it’s much more complicated than making an http request and reading the results from the body. You don’t spin up a browser, much less the full GUI, unless it’s a last resort.
Well, very much yes and no to that claim. Sure, for someone who’s comfortable in the shell, the first step is lighter with curl or wget. But the next step – parsing — is a lot less obvious. And it’s all much more likely to fail on websites that assume a (logged-in?) user in a browser, accepting cookies, executing JS, and so on.