Even getting the data to that machine presents a problem that seems to undermine the value of it entirely: As you mentioned, the real value in this is linear processing of large sets of data, but the storage is ephemeral so your process has to be some variation of firing this instance up, copy TBs of data to the machine, and then do linear processing. Given that you have to get the data there, the value of the high aggregated gross-throughput seems secondary -- just stream process it, etc.
I'm having a tough time seeing where this type of instance fits.
Data Pipeline looks like a fine orchestration service, but it's not going to ingest 48 TB of data any faster than you can do it yourself. Which is probably not that fast.
Comments
Even getting the data to that machine presents a problem that seems to undermine the value of it entirely: As you mentioned, the real value in this is linear processing of large sets of data, but the storage is ephemeral so your process has to be some variation of firing this instance up, copy TBs of data to the machine, and then do linear processing. Given that you have to get the data there, the value of the high aggregated gross-throughput seems secondary -- just stream process it, etc.
I'm having a tough time seeing where this type of instance fits.
That's why AWS Data Pipeline came along. http://aws.amazon.com/datapipeline/
Data Pipeline looks like a fine orchestration service, but it's not going to ingest 48 TB of data any faster than you can do it yourself. Which is probably not that fast.
maybe + http://aws.amazon.com/importexport/