Comment on Building blocks of a scalable webcrawler.parentComments−toumhi15yscrapy (http://scrapy.org/) is a well-documented and open source python scraping framework that I've used in a couple of projects.−rb2k_15yIndeed, seems like a great framework.Considering the timespan of the project, I had to rely on something I'm pretty ok at (Ruby), but I remember hitting a lot of posts about scrapy on the way
Comments
scrapy (http://scrapy.org/) is a well-documented and open source python scraping framework that I've used in a couple of projects.
Indeed, seems like a great framework.
Considering the timespan of the project, I had to rely on something I'm pretty ok at (Ruby), but I remember hitting a lot of posts about scrapy on the way