It's rare to see a comment on HN that misunderstands basic distributed systems concepts. MapReduce the paper has nothing to do with a database. You're likely conflating the fact that to achieve fault tolerant distributed computation, hadoop and hadoop like systems use a database like filesystem.
However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa. That's like saying "wow linkedin made kafka why do we need a webserver too". Those two technologies are only related in the loosest sense.
You're likely conflating the fact that to achieve fault tolerant distributed computation, hadoop and hadoop like systems use a database like filesystem.
Yes, I specifically mentioned non-Google users adopting Hadoop, since it encompassed both a MapReduce implementation and supporting infrastructure.
Once on the bandwagon inspired by the MapReduce paper, many orgs didn't just use MapReduce itself for parallelized batch analytic purposes, but also HBase and Hive and other stuff with actual longer term state atop HDFS, YARN, etc.
However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa.
The marketing and sales teams of HortonWorks, Cloudera, etc certainly sold Hadoop platforms, related Apache projects, and "MapReduce" (as a broad brand name for all this, not the specific technical concept) as replacements for databases, broadly speaking. It's that culture that was a bit shocked when Spanner was unveiled.
The reason spanner is a big deal is because it provides relational semantics including cross table transactions. That is why spanner is interesting.
No one, not even the most ambitious and clueless market folks at Cloudera, has ever sold Hadoop and MR type technologies as a replacement for RDBMs. Spanner can be used as a replacement. These two technologies don't solve the same problem.
You still need MR to do analytics at scale on spanner!
Comments
It's rare to see a comment on HN that misunderstands basic distributed systems concepts. MapReduce the paper has nothing to do with a database. You're likely conflating the fact that to achieve fault tolerant distributed computation, hadoop and hadoop like systems use a database like filesystem.
However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa. That's like saying "wow linkedin made kafka why do we need a webserver too". Those two technologies are only related in the loosest sense.
Yes, I specifically mentioned non-Google users adopting Hadoop, since it encompassed both a MapReduce implementation and supporting infrastructure.
Once on the bandwagon inspired by the MapReduce paper, many orgs didn't just use MapReduce itself for parallelized batch analytic purposes, but also HBase and Hive and other stuff with actual longer term state atop HDFS, YARN, etc.
The marketing and sales teams of HortonWorks, Cloudera, etc certainly sold Hadoop platforms, related Apache projects, and "MapReduce" (as a broad brand name for all this, not the specific technical concept) as replacements for databases, broadly speaking. It's that culture that was a bit shocked when Spanner was unveiled.
The reason spanner is a big deal is because it provides relational semantics including cross table transactions. That is why spanner is interesting.
No one, not even the most ambitious and clueless market folks at Cloudera, has ever sold Hadoop and MR type technologies as a replacement for RDBMs. Spanner can be used as a replacement. These two technologies don't solve the same problem.
You still need MR to do analytics at scale on spanner!