Let's pessimistically assume you have to read all information associated with each file from disk. I suspect this sort of query is IO bound since processing is minimal and assuming we can mostly avoid fragmentation on disk we could feasibly read data at 100MB/s with a consumer level magnetic drive. The article says the data set is 300GB so we could feasibly get this done in an hour.
Comments
Let's pessimistically assume you have to read all information associated with each file from disk. I suspect this sort of query is IO bound since processing is minimal and assuming we can mostly avoid fragmentation on disk we could feasibly read data at 100MB/s with a consumer level magnetic drive. The article says the data set is 300GB so we could feasibly get this done in an hour.