Hi I work in the log importation services business.
From my perspective the main hurdle to log aggregation/correlation is not scalability. If splunk doesn't cut it for your performance needs, you have probably hit the price point to where you can afford a loglogic or similar appliance.
Instead the barrier to entry is in the number of applications supported by a particular log archival product, and the ability to correlate across the different applications.
As I'm sure you know at this point, adding support for log types is a painstaking task. Most vendors punt on this and tell customers to do it themselves.
If there is a niche available to you as a startup I would think that it would be in offering a very low turnaround time in supporting new log types. For example: give us some log sources and we'll support and categorize your logs with our service.
As for running in the cloud on large datasets, I think you'll find that most customers are not going to want to double or triple their outgoing bandwidth -- In addition to concerns from a security compliance standpoint.
That being said, good luck in your venture. Logging is a mess, and could certainly use some clean up. :)
Comments
Hi I work in the log importation services business.
From my perspective the main hurdle to log aggregation/correlation is not scalability. If splunk doesn't cut it for your performance needs, you have probably hit the price point to where you can afford a loglogic or similar appliance.
Instead the barrier to entry is in the number of applications supported by a particular log archival product, and the ability to correlate across the different applications.
As I'm sure you know at this point, adding support for log types is a painstaking task. Most vendors punt on this and tell customers to do it themselves.
If there is a niche available to you as a startup I would think that it would be in offering a very low turnaround time in supporting new log types. For example: give us some log sources and we'll support and categorize your logs with our service.
As for running in the cloud on large datasets, I think you'll find that most customers are not going to want to double or triple their outgoing bandwidth -- In addition to concerns from a security compliance standpoint.
That being said, good luck in your venture. Logging is a mess, and could certainly use some clean up. :)