Comment on Show HN: Crawlee for Python – a web scraping and browser automation libraryComments−Findecanor2yDoes it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags? These are getting more important now, especially in the EU after the DSM directive.−jancurnOP2yNot yet, but it’s on the roadmap
Comments
Does it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags? These are getting more important now, especially in the EU after the DSM directive.
Not yet, but it’s on the roadmap