The one that is really irksome is the COMPACT STORAGE one, as it is explicitly disrecommended and deprecated by the Cassandra documentation. I have not yet gotten around to using Cassandra in any production environment, nor have I done any large-scale tests, and it has been a while since I delved deeply into the data model, but I remember looking at this part of the storage system and thinking it was "wrong", but wondered if I was just misunderstanding things, given how clearly and strongly it was discouraged. Flipping through the presentation you just linked, they are using all the new features and there are no slides that mention compact storage. "30x", as reported in this article, is clearly non-trivial: what is your take?
The GP is right, I started using Cassandra right before they got a hardon for "CQL" and the docs used to explicitly layout the data model when using the thrift interface. (The Thrift data model is basically what you get when you use COMPACT STORAGE, we still using CQL). Simply put, I bet the 30x increase in performance is not because they used COMPACT STORAGE, and is because Map/collections have terrible performance, and COMPACT STORAGE forced them describe the data the "right" way.
Instead of using a map (or COMPACT STORAGE), they should have defined their schema upfront (one of the limitations brought on by not-SQL). However if they didn't want to do that then COMPACT STORAGE is obviously a better solution than using a Map.
To answer your question about COMPACT STORAGE, standard CQL basically does the work for you (in terms of parsing the "xsv") but you have you define the schema upfront (as in, you have know all the map keys before hand). The reason they tell you not to use COMPACT STORAGE is for those cases where you don't need a dynamic schema, using COMPACT STORAGE doesn't really get you anything.
Lastly, IMO, I wouldn't touch CQL Collections or COMPACT STORAGE unless absolutely necessary. If you do need a dynamic schema I would rather encode the data as a msgpack or protobuf blob.
If thats the case, then thats something you should have highlighted that as well. Now that I think about it we decided to store blobs under one column using msgpack for a similar reason (although mine was "we had mongo eat up all our disk space because of field names, so now our code is littered with single character field names"). I've thought about fixing it, but if its the case I guess thats related to CASSANDRA-4175. The map solution makes it come across you weren't quite sure what you were doing.
However, I'm even more surprised at the "slower for queries" part. Maybe I'll do some tests with COMPACT STORAGE.
Might be specific to the Python driver, but it was slower for queries because most of the cputime was being spent decoding column structure/metadata and on type conversions. We also tested msgpack blobs, iirc, but the xsv format was the winner for query perf and for compactness on disk.
Would have gladly gone into more detail, but at 4,000+ words for the blog post already... :)
Hmm, so I tried converting a table we had from a standard table to one with COMPACT STORAGE. The space saving wasn't all that great (100GB -> 80GB), not near 30x.
I stand by my point that the mistake was using CQL maps when you should have just used a defined schema.
I haven't dealt with the ops side of running our clusters, and the details are not fresh in my mind, so I don't remember much about the storage engines. But we encountered surprises on almost any front when taking C* to production.
Given your experience: Have you considered a NewSQL db like MariaDB with one of its modern storage engines? Also Facebook moved from their Cassandra to Hadoop/Hive a long time ago (and the MySQL side was never touched).
We've moved most of our data to redis (which we were using way before cassandra), with a bit of classic MySQL for off-line data, and Redshift for analytics. Right now redis is not using cluster mode, but we'll switch to it eventually. It's a big move, but I find redis more predictable, and for our data sizes (10s of Gs), cheaper to maintain.
Do you use Redis as the master data store? Are you using Redis Cluster, or unclustered?
My concern with Redis is reliability. Redis Cluster has problems with consistency [1], whereas unclustered Redis -- well, it syncs to disk every 10 seconds or so, but even then I'm concerned that the reliability of its on-disk structures haven't been as battle-tested as, say, PostgreSQL. Or has it?
Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes when no fsync is in progress.) but you can only lose one second worth of writes.
Yeah, you can. Its quite nice, the only real problem with it is the reliability of the clustering is subpar.
If you are happy with a Master/Slave setup and occasionally having to deal with data loss due to Master failures [e.g. losing a second or two of data that wasn't replicated], it works nicely.
Just realize I wouldn't use Redis as a long term persisting of data because of issues like this:
the dropping prices of RAM lead us to re-evaluate what we consider "data that fits in RAM". at 10Gs it's just more economic, and even if we go up one order of magnitude, the serving speed of redis still makes it economic for a lot of workloads.
If you moved your data off Caasandra to Redis, I am going to assume you must be using a lot of Lua to maintain some relationships in data in Redis. How is the performance? I have been experimenting with Redis and my lua scripts are long and I have been wondering how they will fare under production level loads.
A Redis Hash is similar enough to Casandra's data structure I doubt they'd need that.
row key = key
column = field
I know Lua is "an option" in Redis but I avoid it like the plague because under production loads, Redis hashes perform and are easy to maintain w/o Lua.
More or less this. Row = HASH key, sorted sets for secondary key indexing, and also for iterating primary key entries. I've built an abstraction layer on top of redis to automate all that and it works rather well.
Comments
The one that is really irksome is the COMPACT STORAGE one, as it is explicitly disrecommended and deprecated by the Cassandra documentation. I have not yet gotten around to using Cassandra in any production environment, nor have I done any large-scale tests, and it has been a while since I delved deeply into the data model, but I remember looking at this part of the storage system and thinking it was "wrong", but wondered if I was just misunderstanding things, given how clearly and strongly it was discouraged. Flipping through the presentation you just linked, they are using all the new features and there are no slides that mention compact storage. "30x", as reported in this article, is clearly non-trivial: what is your take?
The GP is right, I started using Cassandra right before they got a hardon for "CQL" and the docs used to explicitly layout the data model when using the thrift interface. (The Thrift data model is basically what you get when you use COMPACT STORAGE, we still using CQL). Simply put, I bet the 30x increase in performance is not because they used COMPACT STORAGE, and is because Map/collections have terrible performance, and COMPACT STORAGE forced them describe the data the "right" way.
Instead of using a map (or COMPACT STORAGE), they should have defined their schema upfront (one of the limitations brought on by not-SQL). However if they didn't want to do that then COMPACT STORAGE is obviously a better solution than using a Map.
To answer your question about COMPACT STORAGE, standard CQL basically does the work for you (in terms of parsing the "xsv") but you have you define the schema upfront (as in, you have know all the map keys before hand). The reason they tell you not to use COMPACT STORAGE is for those cases where you don't need a dynamic schema, using COMPACT STORAGE doesn't really get you anything.
Lastly, IMO, I wouldn't touch CQL Collections or COMPACT STORAGE unless absolutely necessary. If you do need a dynamic schema I would rather encode the data as a msgpack or protobuf blob.
It wasn't 30x performance improvement; it was 30x less disk space used. Which is nontrivial at their scale.
You said: "they should have defined their schema upfront".
We tried. It was not only less disk efficient, but also slower for queries. Wasn't the result we expected, but alas.
If thats the case, then thats something you should have highlighted that as well. Now that I think about it we decided to store blobs under one column using msgpack for a similar reason (although mine was "we had mongo eat up all our disk space because of field names, so now our code is littered with single character field names"). I've thought about fixing it, but if its the case I guess thats related to CASSANDRA-4175. The map solution makes it come across you weren't quite sure what you were doing.
However, I'm even more surprised at the "slower for queries" part. Maybe I'll do some tests with COMPACT STORAGE.
Might be specific to the Python driver, but it was slower for queries because most of the cputime was being spent decoding column structure/metadata and on type conversions. We also tested msgpack blobs, iirc, but the xsv format was the winner for query perf and for compactness on disk.
Would have gladly gone into more detail, but at 4,000+ words for the blog post already... :)
Hmm, so I tried converting a table we had from a standard table to one with COMPACT STORAGE. The space saving wasn't all that great (100GB -> 80GB), not near 30x.
I stand by my point that the mistake was using CQL maps when you should have just used a defined schema.
I haven't dealt with the ops side of running our clusters, and the details are not fresh in my mind, so I don't remember much about the storage engines. But we encountered surprises on almost any front when taking C* to production.
Given your experience: Have you considered a NewSQL db like MariaDB with one of its modern storage engines? Also Facebook moved from their Cassandra to Hadoop/Hive a long time ago (and the MySQL side was never touched).
We've moved most of our data to redis (which we were using way before cassandra), with a bit of classic MySQL for off-line data, and Redshift for analytics. Right now redis is not using cluster mode, but we'll switch to it eventually. It's a big move, but I find redis more predictable, and for our data sizes (10s of Gs), cheaper to maintain.
Do you use Redis as the master data store? Are you using Redis Cluster, or unclustered?
My concern with Redis is reliability. Redis Cluster has problems with consistency [1], whereas unclustered Redis -- well, it syncs to disk every 10 seconds or so, but even then I'm concerned that the reliability of its on-disk structures haven't been as battle-tested as, say, PostgreSQL. Or has it?
[1] https://aphyr.com/posts/307-call-me-maybe-redis-redux
Yeah, about that:
http://redis.io/topics/persistence
RDB is what I was thinking of. I didn't know they had added a write-ahead log. It seems you can combine RDB and AOF, which is nice.
Yeah, you can. Its quite nice, the only real problem with it is the reliability of the clustering is subpar.
If you are happy with a Master/Slave setup and occasionally having to deal with data loss due to Master failures [e.g. losing a second or two of data that wasn't replicated], it works nicely.
Just realize I wouldn't use Redis as a long term persisting of data because of issues like this:
https://muut.com/blog/news/april-2014-service-failure.html
Redis is neet if your information fits in RAM, but that's not what Cassandra is for.
the dropping prices of RAM lead us to re-evaluate what we consider "data that fits in RAM". at 10Gs it's just more economic, and even if we go up one order of magnitude, the serving speed of redis still makes it economic for a lot of workloads.
Eh...as long as you are ready and able to shard the dataset, fitting the data in RAM is quite possible for TB sized datasets.
You can easily get commodity servers with 256GB of RAM per node x 10 shards. Cluster or not, as per your use case.
If you moved your data off Caasandra to Redis, I am going to assume you must be using a lot of Lua to maintain some relationships in data in Redis. How is the performance? I have been experimenting with Redis and my lua scripts are long and I have been wondering how they will fare under production level loads.
A Redis Hash is similar enough to Casandra's data structure I doubt they'd need that.
row key = key
column = field
I know Lua is "an option" in Redis but I avoid it like the plague because under production loads, Redis hashes perform and are easy to maintain w/o Lua.
More or less this. Row = HASH key, sorted sets for secondary key indexing, and also for iterating primary key entries. I've built an abstraction layer on top of redis to automate all that and it works rather well.