Thanks for sharing this info! Of all the points you listed I think the most important for me is "MySQL is well understood, which makes looking after it quite easy". Personally I think that our analytics requirements can be satisfied by at least half a dozen different systems out there, so management, optimisation and maintenance will be major factors. For my part, these are my thoughts on the systems we've looked at:
1. Riak: K/V stores are conceptually simple, secondary indexes look nice, and the consensus from RICON was that scaling by adding nodes pretty-much Just Works. However, no-one at RICON seemed to use MapReduce, particularly not for real-time analytics. No SQL-like querying.
2. HBase: Built on popular technologies. Amazon provides it as a service, but altogether it's very complex (a lot of components, and we're not a Java shop) and we have had trouble achieving the performance we need - although it seems from various blogs and books that it must be possible.
3. Hypertable: Still testing it, hard to find much information about it outside of the main site.
4. Postgres: Everyone knows and loves Postgres. Scaling (and performance after scaling) are concerns - we might have to go down the sharding route, and may never be able to store raw events data as it comes in.
Haven't looked at Cassandra yet, but most likely will do that soon.
The trick to hbase performance almost always relies on getting the row key and column qualifier design right. Your keys and qualifiers should be chosen to exploit bulk scans. If you have any specific questions, shoot me an email I can give you some pointers (email in profile).
For scaling postgres, have a look at PL/Proxy. You can basically partition tables across servers without knowledge of the fact that the other servers are even there. However, it doesn't easily cover you for fault tolerance and recovery.
Comments
Thanks for sharing this info! Of all the points you listed I think the most important for me is "MySQL is well understood, which makes looking after it quite easy". Personally I think that our analytics requirements can be satisfied by at least half a dozen different systems out there, so management, optimisation and maintenance will be major factors. For my part, these are my thoughts on the systems we've looked at:
1. Riak: K/V stores are conceptually simple, secondary indexes look nice, and the consensus from RICON was that scaling by adding nodes pretty-much Just Works. However, no-one at RICON seemed to use MapReduce, particularly not for real-time analytics. No SQL-like querying.
2. HBase: Built on popular technologies. Amazon provides it as a service, but altogether it's very complex (a lot of components, and we're not a Java shop) and we have had trouble achieving the performance we need - although it seems from various blogs and books that it must be possible.
3. Hypertable: Still testing it, hard to find much information about it outside of the main site.
4. Postgres: Everyone knows and loves Postgres. Scaling (and performance after scaling) are concerns - we might have to go down the sharding route, and may never be able to store raw events data as it comes in.
Haven't looked at Cassandra yet, but most likely will do that soon.
The trick to hbase performance almost always relies on getting the row key and column qualifier design right. Your keys and qualifiers should be chosen to exploit bulk scans. If you have any specific questions, shoot me an email I can give you some pointers (email in profile).
For scaling postgres, have a look at PL/Proxy. You can basically partition tables across servers without knowledge of the fact that the other servers are even there. However, it doesn't easily cover you for fault tolerance and recovery.