Made something very similar to this in PHP about a year ago. It's buried in the Drupal project called Boost. It's the multi process crawler. It supports up to 8 processes crawling your site getting the cache warm. Gets call on cron & then 15 seconds after drupal's cron finished it starts up all 8 "threads". I've used it for image processing as well. Very stable will run for days if you give it a big enough queue (millions of pages to crawl). http://drupal.org/project/boost
Comments
Made something very similar to this in PHP about a year ago. It's buried in the Drupal project called Boost. It's the multi process crawler. It supports up to 8 processes crawling your site getting the cache warm. Gets call on cron & then 15 seconds after drupal's cron finished it starts up all 8 "threads". I've used it for image processing as well. Very stable will run for days if you give it a big enough queue (millions of pages to crawl). http://drupal.org/project/boost