I don't mean to come off sounding like a mouthpiece for Splunk, but Splunk does work with "cloud" data (hurray for buzz words), it is dead simple to use out of the box and recognizes many special fields like timestamps without any configuration (and also provides extensive configurability when needed) and scales like the distributed monster it is, capable of handling 10-15 MB of data / sec. on a single machines and GBs worth of data per day when scaled out across multiple machines. You should try it before you judge it on hearsay (and I'll maintain that you should also try any Splunk competitor before judging them as well).
I have tried Splunk. It's a great concept and I commend the Splunk team for building such an easy to use, polished product. However we found the query performance to be... ahem... not well geared towards large data sets. This combined with the licensing model meant it wasn't an option for us.
I just wish Yahoo would open source Everest (their multi-PB column store DB based on PostgreSQL) -- this would be ideal for building an open source Splunk competitor.
Interesting. Do you know which version of Splunk you were using? Our latest version has vastly improved our query performance over large datasets.
Re. an open source log indexer: Agree, this is a space that will eventually become dominated by open source tools, used particularly by startups and small businesses. I think most people ignore this use of a MapReduce-like framework because they conceptually understand how it could be used, but 99% of all work is in the implementation, not the idea. And as of yet, I don't believe there has been a specific implementation beyond what companies like Shopify are doing where they add nice GUI tools on top of awk and grep (which admittedly is probably good enough for most people / business on this forum).
For scalability use SenSage (http://www.sensage.com/customers). Their some of their customers gather 200GB-500GB of log data daily and at least one customer has a petabyte of historical log data under management. Query performance over long time ranges is important ... when you discover a new exploit in use, how can you tell how long it's been employed unless you can look at history?
If you have huge datasets, we'd love to hear from you. If even to chat for a few minutes about what your data looks like. Ping me at info@drawntoscale.com -- maybe we can help!
Comments
It's true that LogSearch is similar, but we focus on cloud, analytics, ease-of-use and scalability -- each of which we'ce heard Splunk lacks.
I don't mean to come off sounding like a mouthpiece for Splunk, but Splunk does work with "cloud" data (hurray for buzz words), it is dead simple to use out of the box and recognizes many special fields like timestamps without any configuration (and also provides extensive configurability when needed) and scales like the distributed monster it is, capable of handling 10-15 MB of data / sec. on a single machines and GBs worth of data per day when scaled out across multiple machines. You should try it before you judge it on hearsay (and I'll maintain that you should also try any Splunk competitor before judging them as well).
I have tried Splunk. It's a great concept and I commend the Splunk team for building such an easy to use, polished product. However we found the query performance to be... ahem... not well geared towards large data sets. This combined with the licensing model meant it wasn't an option for us.
I just wish Yahoo would open source Everest (their multi-PB column store DB based on PostgreSQL) -- this would be ideal for building an open source Splunk competitor.
Interesting. Do you know which version of Splunk you were using? Our latest version has vastly improved our query performance over large datasets.
Re. an open source log indexer: Agree, this is a space that will eventually become dominated by open source tools, used particularly by startups and small businesses. I think most people ignore this use of a MapReduce-like framework because they conceptually understand how it could be used, but 99% of all work is in the implementation, not the idea. And as of yet, I don't believe there has been a specific implementation beyond what companies like Shopify are doing where they add nice GUI tools on top of awk and grep (which admittedly is probably good enough for most people / business on this forum).
For scalability use SenSage (http://www.sensage.com/customers). Their some of their customers gather 200GB-500GB of log data daily and at least one customer has a petabyte of historical log data under management. Query performance over long time ranges is important ... when you discover a new exploit in use, how can you tell how long it's been employed unless you can look at history?
Yes, Everest is cool stuff.
If you have huge datasets, we'd love to hear from you. If even to chat for a few minutes about what your data looks like. Ping me at info@drawntoscale.com -- maybe we can help!
I miss spoke. Splunk handles multiple TBs per day easily in a distributed environment.
We have tried them :) And so have several of our customers.