Comment on Who needs to scrape millions of pages, or monitor them?parentComments−calufaOP13yIt will basically go through all the links it finds, that can be millions of links. You can also tell it to ignore certain links using regular expressions or via tales java apis.−calufaOP13yHere is a sample code of a 1 depth scrape:https://github.com/calufa/tales-templates/blob/master/core/s...This is the api call to start the scraper on twitter:http://localhost:8080/start?process=tales.scrapers.LoopScrap... -template tales.templates.FirstDepthTemplate -threads 2 -namespace com_twitter -baseURL twitter.com
Comments
It will basically go through all the links it finds, that can be millions of links. You can also tell it to ignore certain links using regular expressions or via tales java apis.
Here is a sample code of a 1 depth scrape:
https://github.com/calufa/tales-templates/blob/master/core/s...
This is the api call to start the scraper on twitter:
http://localhost:8080/start?process=tales.scrapers.LoopScrap... -template tales.templates.FirstDepthTemplate -threads 2 -namespace com_twitter -baseURL twitter.com