The crawler currently runs on a single large EC2 instance.
I could see myself trying to use a bunch of EC2 micro instances instead and then use Riak + Riak Search.
I actually tried putting a dump of the data into Riak and it seemed to hold up pretty well on my macbook.
Another problem was the fact that Riak didn't allow me to do server-side increments on the "incoming links" counter which mysql, mongodb or redis allowed. However, I think that this is something that could be solved using Redis as a caching layer.
I have to admit that I would love to use Riak for something just because it seems to be a really slick piece of software, so it's hard to stay objective :)
Comments
The crawler currently runs on a single large EC2 instance. I could see myself trying to use a bunch of EC2 micro instances instead and then use Riak + Riak Search.
I actually tried putting a dump of the data into Riak and it seemed to hold up pretty well on my macbook.
Another problem was the fact that Riak didn't allow me to do server-side increments on the "incoming links" counter which mysql, mongodb or redis allowed. However, I think that this is something that could be solved using Redis as a caching layer.
I have to admit that I would love to use Riak for something just because it seems to be a really slick piece of software, so it's hard to stay objective :)