What you really want for time-series data is a column db such as cassandra (or vertica etc.), perhaps HBase, perhaps a RDBMS, or perhaps a plain old log-file.
What you most definitely don't want is Microsoft Access or MongoDB.
Thinking about it, MS Access might still work to a degree.
As long as it all the data for all metrics fits in RAM, it will work great in MongoDB. Of course, as soon as it doesn't, you're totally hosed.
To expand on the parent, a column store like HBase or Cassandra is perfect as each row can represent a timeslice and then each column can represent a single event or record within that timeslice. As the row gets evicted from it's initial storage in memory, it will be written to disk contiguously, in sorted order, and batch reads of this data is sequential.
It is possible to use MongoDB to store a timeslice as a document, but it is not designed to scale out to store very large numbers of columns within a single document.
happily, this is not a synchronous application with tens of thousands of concurrent users, so "totally hosed" for us may have a very different definition.
What do you mean by "very large numbers of columns" ? I've seen some mongo users with very rich, i.e. large, document models.
1,000s, 10,000s, 100,000s, millions. MongoDB columns are designed for serialization of rich documents, not store unbounded ranges of data values.
Even without tens of thousands of concurrent users, you'll eventually run into a deeply critical performance wall when MongoDB starts reading from disk. It's really best to think of it as an in-memory database.
Vertica was not free, HBase was really heavyweight, & I knew i didn't want RDBMS. The sensor data itself is plain csvs ;)
Cassandra still does look interesting, although it looked like it would take me much longer to get it going; perhaps it will get rewritten. (The mongo solution only took a week to get something usable.)
Curious to hear why you thought it would take longer to get going with Cassandra. A single-node Cassandra cluster can be up with a 1-line command. Was it the availability of clients or interface abstractions? I understand that tutorial & documentation is not as readily available, so that's understandable.
It wasn't the ease of install or simplest-case-deployment, that's for sure. I did install it and kick the tires.
This was around the end of 2009, and i don't think there were many clients available (i see pycassa dates to 2010-04).
again, ease of development is much more important to us than performance; it'll probably be a long, long time before our db engine is the choke point, at which point we'll have resources to use a heavier tool.
Is it just about the number of dimensions? Seems like you'd get decent performance with mongodb if each timeseries is a collection, and each entry was another document?
Comments
Aw, this was physically painful to skim.
What you really want for time-series data is a column db such as cassandra (or vertica etc.), perhaps HBase, perhaps a RDBMS, or perhaps a plain old log-file.
What you most definitely don't want is Microsoft Access or MongoDB. Thinking about it, MS Access might still work to a degree.
As long as it all the data for all metrics fits in RAM, it will work great in MongoDB. Of course, as soon as it doesn't, you're totally hosed.
To expand on the parent, a column store like HBase or Cassandra is perfect as each row can represent a timeslice and then each column can represent a single event or record within that timeslice. As the row gets evicted from it's initial storage in memory, it will be written to disk contiguously, in sorted order, and batch reads of this data is sequential.
It is possible to use MongoDB to store a timeslice as a document, but it is not designed to scale out to store very large numbers of columns within a single document.
MongoDB needs indexes to fit in memory, not the entire dataset. This is an important distinction.
happily, this is not a synchronous application with tens of thousands of concurrent users, so "totally hosed" for us may have a very different definition.
What do you mean by "very large numbers of columns" ? I've seen some mongo users with very rich, i.e. large, document models.
1,000s, 10,000s, 100,000s, millions. MongoDB columns are designed for serialization of rich documents, not store unbounded ranges of data values.
Even without tens of thousands of concurrent users, you'll eventually run into a deeply critical performance wall when MongoDB starts reading from disk. It's really best to think of it as an in-memory database.
glad you didn't like it.
Vertica was not free, HBase was really heavyweight, & I knew i didn't want RDBMS. The sensor data itself is plain csvs ;)
Cassandra still does look interesting, although it looked like it would take me much longer to get it going; perhaps it will get rewritten. (The mongo solution only took a week to get something usable.)
Curious to hear why you thought it would take longer to get going with Cassandra. A single-node Cassandra cluster can be up with a 1-line command. Was it the availability of clients or interface abstractions? I understand that tutorial & documentation is not as readily available, so that's understandable.
It wasn't the ease of install or simplest-case-deployment, that's for sure. I did install it and kick the tires.
This was around the end of 2009, and i don't think there were many clients available (i see pycassa dates to 2010-04).
again, ease of development is much more important to us than performance; it'll probably be a long, long time before our db engine is the choke point, at which point we'll have resources to use a heavier tool.
This makes sense, the landscape was very different. Cassandra wasn't something I would have used back then either.
Is it just about the number of dimensions? Seems like you'd get decent performance with mongodb if each timeseries is a collection, and each entry was another document?