Skip to content

Comment on Building blocks of a scalable webcrawler.parent

Comments

After having written the thesis and thought about that stuff for another few weeks, my résumé would be:

- Use asynchronous I/O to maximize single-node speed (twisted should be a good choice for python). It might be strange in the beginning, but it usually makes up for it, especially with languages that aren't good at threading (ruby, python, ...).

- Redis is awesome! Fast, functional, beautiful :)

- Riak seems to be a great distributed datastore if you really have to scale over multiple nodes.

- Solr or Sphinx are just better optimized than most datastores when it comes to fulltext-search

- Take a day to look at graph databases (I'm still not 100% sure if I could have used one for my use cases)

Thanks for the tips! I really appreciate it. I'll check these out. All getting very exciting for my Christmas project!

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.