Comment on Building blocks of a scalable webcrawler.parentComments−rb2k_15yDo you remember how fast the "toy" was? (pages/second, domains/s, ...) :)−sriramk15yNot really but given the terrible hardware/network connectivity , wouldnt have made much sense now.Because of this thread, I looked through my old backups and I actually still have the code. Should get it working again sometime−sandGorgon15yare you gonna put up your code ?It would be interesting to see how to think through building a crawler (as opposed to downloading Nutch and trying to grok it)
Comments
Do you remember how fast the "toy" was? (pages/second, domains/s, ...) :)
Not really but given the terrible hardware/network connectivity , wouldnt have made much sense now.
Because of this thread, I looked through my old backups and I actually still have the code. Should get it working again sometime
are you gonna put up your code ?
It would be interesting to see how to think through building a crawler (as opposed to downloading Nutch and trying to grok it)