"However, none of them seemed widely-used enough by large sites to inspire confidence. In the tests we read about and ran ourselves, none of the projects were stable or battle-tested enough for our needs"
Ok, just some hour ago I released the beta-3 of Redis (http://code.google.com/p/redis/ if you care) and I'm near to feature-freeze with exactly with this goal. To make it rock solid (I'm going to use it in my startup's web stuff with a lot of users/month, so I care about stability).
The question is: what's in your opinion the right path to make a system like Redis stable and reliable for the real world usage? What to publish on the site in order to inspire a good feeling about stability? Thanks
Not to be a naysayer, but I don't know that there's a lot you can do except run it yourself on a large, popular site that people have heard of. Companies like FriendFeed will use memcached cause LiveJournal/Facebook use it, they'll use MySQL because just about every web startup uses it, but no matter how awesome a project is, they're not going to use something complex that a few developers or some other startup wrote that hasn't been battle-tested on a large, well-known site. It's too risky compared to writing another one yourself that you understand.
1) Take Redis simple enough so that a single developer can understand the implementation in little time.
2) Run Redis on my own large sites. The problem is that while this sites are really large they are famous only in Italy. One example is http://oknotizie.alice.it
3) In the future when Redis will be stable and some little startup will start using it for real work try to collect success stories and write it in the front page of the project.
I've been evaluating databases recently. It doesn't matter if the site is not well-known in the US as long as you can post traffic numbers. TokyoTyrant is Japanese, after all.
Also these help:
* able to handle large (>200GB) datasets
* client libraries for top N languages
* easy way to write client libs (eg simple protocol, a C library, etc)
* connection pooling and/or cheap connections
* easy to install on Mac, Linux and Windows development machines
* repl (this is not much of a problem with Python & Ruby)
* stable dump / restore format
One problem for you is that MySQL plus serialize() basically does these already, with 10+ years of testing on top. Your system has to do a lot more to make it worth the risk.
Scale enough in which way? Read traffic? Write traffic? Locking? If you use MySQL as a btree it is pretty fast and consistent. I gather from this post that a lot of companies started in the last 18 months are doing this. It's a hack but a remarkably useful one. Good luck! I will keep my eye on redis.
Probably it just depends on the dataset. The issue we have with our service (http://lloogg.com) is that you need to take the last N items of logs for every site. To ask for the latest M items should be fast. To push a new log line on the list should be fast. Every kind of MySQL configuration we tried was unable to reach the 10k writes/second we reach with Redis. Obviously. Even when you use MySQL as a btree implementation you get a lot of overhead. Starting from the protocol and the format of the statements, for example.
The idea to encode things with json or other formats in a blog text is just a ugly hack. People are using this because they are desperate, not because is good computer science. They started with mysql, know mysql, hacked with mysql. Clearly will try to fix their site with MySQL.
The json+blob can work as long as the data that's stored in this fields is trivial to serialize-deserialize. What about having a 10000 elements list in every blob and at every page view you need to append an element?
So: great hack, you found a way to work with the tools you have, but this does not mean in any way that fast key-value persistent DBs don't have something to say into the web-scale theater.
I added consistent ring hashing and a rediscloud class to the ruby client library today. so you can now use a cluster of redis servers and have the client use ring hashing to determine the server for a given key.
it's higher level. Not a plain key value stuff. For instance as value you can have a list or a set, push/pop elements, ask the server for all the keys matching a given glob style pattern and so on. Most of this operations are atomic in order to make sure there are no race conditions.
Basically the long term goal is to have something between a relational DB and a key-value DB. Not all the higher level features must be killed in order to be scalable.
Has anyone here got TokyoTyrant working under Windows? I found a static library binary for Windows, but the header files need to compile are either not there, or the code references some bizarre POSIX stuff which isn't easy to hack into working under Windows.
Comments
"However, none of them seemed widely-used enough by large sites to inspire confidence. In the tests we read about and ran ourselves, none of the projects were stable or battle-tested enough for our needs"
Ok, just some hour ago I released the beta-3 of Redis (http://code.google.com/p/redis/ if you care) and I'm near to feature-freeze with exactly with this goal. To make it rock solid (I'm going to use it in my startup's web stuff with a lot of users/month, so I care about stability).
The question is: what's in your opinion the right path to make a system like Redis stable and reliable for the real world usage? What to publish on the site in order to inspire a good feeling about stability? Thanks
Not to be a naysayer, but I don't know that there's a lot you can do except run it yourself on a large, popular site that people have heard of. Companies like FriendFeed will use memcached cause LiveJournal/Facebook use it, they'll use MySQL because just about every web startup uses it, but no matter how awesome a project is, they're not going to use something complex that a few developers or some other startup wrote that hasn't been battle-tested on a large, well-known site. It's too risky compared to writing another one yourself that you understand.
ok this sounds like I can have this goals:
1) Take Redis simple enough so that a single developer can understand the implementation in little time.
2) Run Redis on my own large sites. The problem is that while this sites are really large they are famous only in Italy. One example is http://oknotizie.alice.it
3) In the future when Redis will be stable and some little startup will start using it for real work try to collect success stories and write it in the front page of the project.
I've been evaluating databases recently. It doesn't matter if the site is not well-known in the US as long as you can post traffic numbers. TokyoTyrant is Japanese, after all.
Also these help:
One problem for you is that MySQL plus serialize() basically does these already, with 10+ years of testing on top. Your system has to do a lot more to make it worth the risk.Agreed on most of the points... about MySQL, I developed Redis just because MySQL does not scale enough with given kind of datasets :)
Scale enough in which way? Read traffic? Write traffic? Locking? If you use MySQL as a btree it is pretty fast and consistent. I gather from this post that a lot of companies started in the last 18 months are doing this. It's a hack but a remarkably useful one. Good luck! I will keep my eye on redis.
Probably it just depends on the dataset. The issue we have with our service (http://lloogg.com) is that you need to take the last N items of logs for every site. To ask for the latest M items should be fast. To push a new log line on the list should be fast. Every kind of MySQL configuration we tried was unable to reach the 10k writes/second we reach with Redis. Obviously. Even when you use MySQL as a btree implementation you get a lot of overhead. Starting from the protocol and the format of the statements, for example.
The idea to encode things with json or other formats in a blog text is just a ugly hack. People are using this because they are desperate, not because is good computer science. They started with mysql, know mysql, hacked with mysql. Clearly will try to fix their site with MySQL.
The json+blob can work as long as the data that's stored in this fields is trivial to serialize-deserialize. What about having a 10000 elements list in every blob and at every page view you need to append an element?
So: great hack, you found a way to work with the tools you have, but this does not mean in any way that fast key-value persistent DBs don't have something to say into the web-scale theater.
I added consistent ring hashing and a rediscloud class to the ruby client library today. so you can now use a cluster of redis servers and have the client use ring hashing to determine the server for a given key.
How is redis different from TokyoTyrant?
it's higher level. Not a plain key value stuff. For instance as value you can have a list or a set, push/pop elements, ask the server for all the keys matching a given glob style pattern and so on. Most of this operations are atomic in order to make sure there are no race conditions.
You can read more about the difference between Redis and other key-value stores here: http://code.google.com/p/redis/wiki/FAQ
Basically the long term goal is to have something between a relational DB and a key-value DB. Not all the higher level features must be killed in order to be scalable.
Has anyone here got TokyoTyrant working under Windows? I found a static library binary for Windows, but the header files need to compile are either not there, or the code references some bizarre POSIX stuff which isn't easy to hack into working under Windows.