Yeah it’s silly, even when Spark is writing unstructured logs, that doesn’t mean that you can‘t parse them after-the-fact and store them in a structured way. Even if it doesn’t work for 100% of the cases, it’s very easy to achieve for 99% of them, in which case you’ll still keep a “raw_message” column which you can query as text.
Next up: Uber discovers column oriented databases are more efficient for data warehouses.
Comments
Yeah it’s silly, even when Spark is writing unstructured logs, that doesn’t mean that you can‘t parse them after-the-fact and store them in a structured way. Even if it doesn’t work for 100% of the cases, it’s very easy to achieve for 99% of them, in which case you’ll still keep a “raw_message” column which you can query as text.
Next up: Uber discovers column oriented databases are more efficient for data warehouses.