Also not wanting to sound like a mouthpiece, but choice of log management system is a very interesting topic to me... scalability is only one concern and may not trump everything. We actually looked at both LogLogic and Sensage during our initial evaluation both of which scaled better than Splunk did at the time. My memory of the evaluations is dim (2+ years ago), but I recall that both products seemed more brittle in our environment (education), where we have very little control over incoming log formats due to somewhat chaotic organizational structure and infrastructure. Splunk seemed more able to accept unstructured or unexpected data without complaint and to deal with the parsing at search time rather than index time, and was a more natural fit. The other decision points/tradeoffs we ended up making were a) no additional licensing cost for Splunk agents, b) prioritization of real-time perusal for troubleshooting purposes over long-term reporting and canned reports. Sensage excels at the latter since it amortizes the cost of reporting by caching periodic results, and provides a large set of prebuilt reports, if I recall correctly. Splunk's "summary indexing" serves sort of the same purpose as the cachign feature of Sensage and has been improved in v4.0. And c) no additional cost for users (we have over 50 people looking at data indexed by Splunk). The "search" metaphor is also very easy for people to grasp which was an added benefit of the Splunk GUI.
My sense has been that at least the low end of the market is increasingly preferring a search-oriented log management architecture versus a database-backed, query-heavy architecture because organizations are familiar with the search metaphor, the overhead of managing a search index is less than the cost of database administration, and the unsophisticated use case (i.e., free-text search rather than advanced query syntax) is increasingly common. Smaller organizations also rarely have the maturity to deal with logging systematically since it requires a pretty systematic approach to infrastructure. That said, the tradeoffs we made may or may not apply in a terabytes-per-day environment, I unfortunately can't speak to it directly.
Comments
Or you can use a tool that has already scaled for ages and has been in production since 2002. SenSage ... http://www.sensage.com/customers.
Also not wanting to sound like a mouthpiece, but choice of log management system is a very interesting topic to me... scalability is only one concern and may not trump everything. We actually looked at both LogLogic and Sensage during our initial evaluation both of which scaled better than Splunk did at the time. My memory of the evaluations is dim (2+ years ago), but I recall that both products seemed more brittle in our environment (education), where we have very little control over incoming log formats due to somewhat chaotic organizational structure and infrastructure. Splunk seemed more able to accept unstructured or unexpected data without complaint and to deal with the parsing at search time rather than index time, and was a more natural fit. The other decision points/tradeoffs we ended up making were a) no additional licensing cost for Splunk agents, b) prioritization of real-time perusal for troubleshooting purposes over long-term reporting and canned reports. Sensage excels at the latter since it amortizes the cost of reporting by caching periodic results, and provides a large set of prebuilt reports, if I recall correctly. Splunk's "summary indexing" serves sort of the same purpose as the cachign feature of Sensage and has been improved in v4.0. And c) no additional cost for users (we have over 50 people looking at data indexed by Splunk). The "search" metaphor is also very easy for people to grasp which was an added benefit of the Splunk GUI.
My sense has been that at least the low end of the market is increasingly preferring a search-oriented log management architecture versus a database-backed, query-heavy architecture because organizations are familiar with the search metaphor, the overhead of managing a search index is less than the cost of database administration, and the unsophisticated use case (i.e., free-text search rather than advanced query syntax) is increasingly common. Smaller organizations also rarely have the maturity to deal with logging systematically since it requires a pretty systematic approach to infrastructure. That said, the tradeoffs we made may or may not apply in a terabytes-per-day environment, I unfortunately can't speak to it directly.