Off-topic, but do you actually run ad-hoc SparkSQL queries on the whole dataset sometimes? Are the logs actually stored as text files on disk? How long does such a query take and/or how many racks of machines do you need for that? Should be more than 1 million drive-seconds just to read the data from disk, right?