FTA: This means in order to find what you want, you need to explore the data: you need to search it. The only tool most of us can use is grep
Um... what? Have the authors gotten stuck in a time vortex and been dropped off before, y'know, awk? Much less perl, or any of those new-fangled toys.
I mean: writing scripts to do log analysis is a pretty fundamental problem for server-side development, and lots of very smart people have spend the last two^H^H^Hthree decades working on tools to address the issue.
I don't even see how this (indexing the entries across a Hadoop cluster) is all that useful. In general, you don't do log analysis by asking "give me all the entries that match this pattern", you do it by walking them in order and extracting one or two fields from each line and building some kind of result data structure. This thing would be fine if you were asking for all the logs messages that mentioned "coffee", I guess. But what if you wanted a histogram of hit counts per page per day-of-week?
Thanks for providing this valuable perspective. I sort of group grep/Perl scripts/everything else together as manual processes. What I was getting at is the whole "roll your own" scripts is a royal pain in the ass.
For analytics, you're right, search is only part of the equation. That's why we make MapReduce easy to use on a cluster. You can write Pig or Hive scr
We also have templates for common data formats (and ways to roll your own) so you can turn unstructured log text into structured data, so that a histogram of hit counts per day-of-week is just a few lines of a script (or maybe even a search).
Comments
FTA: This means in order to find what you want, you need to explore the data: you need to search it. The only tool most of us can use is grep
Um... what? Have the authors gotten stuck in a time vortex and been dropped off before, y'know, awk? Much less perl, or any of those new-fangled toys.
I mean: writing scripts to do log analysis is a pretty fundamental problem for server-side development, and lots of very smart people have spend the last two^H^H^Hthree decades working on tools to address the issue.
I don't even see how this (indexing the entries across a Hadoop cluster) is all that useful. In general, you don't do log analysis by asking "give me all the entries that match this pattern", you do it by walking them in order and extracting one or two fields from each line and building some kind of result data structure. This thing would be fine if you were asking for all the logs messages that mentioned "coffee", I guess. But what if you wanted a histogram of hit counts per page per day-of-week?
Thanks for providing this valuable perspective. I sort of group grep/Perl scripts/everything else together as manual processes. What I was getting at is the whole "roll your own" scripts is a royal pain in the ass.
For analytics, you're right, search is only part of the equation. That's why we make MapReduce easy to use on a cluster. You can write Pig or Hive scr
We also have templates for common data formats (and ways to roll your own) so you can turn unstructured log text into structured data, so that a histogram of hit counts per day-of-week is just a few lines of a script (or maybe even a search).
Looks like I got cut off in the middle. *You can write Pig or Hive scripts to generate interesting analytics.