can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it?
Completely agree with this sentiment.
I just spent the last couple of months developing a chrome extension, but recently also did an unrleated web scraping project where I looked into all the common tools like beautiful soup, selenium, playwright, pupeteer, etc, etc.
All of these tools were needlessly complicated and I was having a ton of trouble with sites that required authentication. I then realized it would be way easier to write some javascript and paste it in my browser to do the scraping. Worked like a charm!
I feel that when it has been set up, it's very straightforward to use.
Maybe in contrast to other solutions you posted? Not sure about that though; having only brief experiences with both, Playwright seems like an improved Cypress to me.
Comments
Completely agree with this sentiment.
I just spent the last couple of months developing a chrome extension, but recently also did an unrleated web scraping project where I looked into all the common tools like beautiful soup, selenium, playwright, pupeteer, etc, etc.
All of these tools were needlessly complicated and I was having a ton of trouble with sites that required authentication. I then realized it would be way easier to write some javascript and paste it in my browser to do the scraping. Worked like a charm!
Is Playwright really that complicated?
I feel that when it has been set up, it's very straightforward to use.
Maybe in contrast to other solutions you posted? Not sure about that though; having only brief experiences with both, Playwright seems like an improved Cypress to me.
You might like Tamper monkey. You can add a button to kick it off or whatever your heart desires.
Tampermoney also works around CORs issues with relative ease.
Userscripts for the win. Consider Violentmonkey over Tampermonkey though.
Why?
Probably because it looks to be closed source these days.
https://github.com/Tampermonkey/tampermonkey
Open source
I love Tamper monkey! I've never made my own scripts though - just used other peoples'.