Worth noting that it sounds like hs1.8xlarge is built on magnetic disks (24 2TB HDs - edit: originally put 1TB), each reading some 100MB/sec, yielding the theoretical max of 2.4GB/s in a RAID-0 configuration. No one actually uses disks in such a fashion, and gross throughput of magnetic drives has seldom been of much utility (hence the strong demand for SSDs. Random IO matches the vast majority of workloads more appropriately).
Just caveats. This doesn't look like a terribly interesting option.
Good point, but their blog post does give a few possible uses: "Storage instances are ideal for data-intensive applications including Hadoop workloads, log processing and data warehousing, and parallel file systems to process and analyze large data sets in the AWS Cloud". If your code fits the pattern, a high-storage instance may fit well.
Of course, getting that much data into and out of the cloud is its own problem.
Even getting the data to that machine presents a problem that seems to undermine the value of it entirely: As you mentioned, the real value in this is linear processing of large sets of data, but the storage is ephemeral so your process has to be some variation of firing this instance up, copy TBs of data to the machine, and then do linear processing. Given that you have to get the data there, the value of the high aggregated gross-throughput seems secondary -- just stream process it, etc.
I'm having a tough time seeing where this type of instance fits.
Data Pipeline looks like a fine orchestration service, but it's not going to ingest 48 TB of data any faster than you can do it yourself. Which is probably not that fast.
Not sure what you mean... there are quite a few data storage & processing tools out there that use a log-structured on-disk storage that were designed to read/write sequentially. These could potentially take full advantage of these instances.
Many other solution requiring random IO is indeed a fact, but I still think there are systems that could benefit.
I mean that no one ever would configure 24 disks as RAID-0 -- the probability of a data-loss failure (which may not be a problem from the data-loss side, but is from the process continuity side) becomes incredibly high. Best case most people would arrange it as RAID-10, instantly dropping the performance by half.
Comments
Worth noting that it sounds like hs1.8xlarge is built on magnetic disks (24 2TB HDs - edit: originally put 1TB), each reading some 100MB/sec, yielding the theoretical max of 2.4GB/s in a RAID-0 configuration. No one actually uses disks in such a fashion, and gross throughput of magnetic drives has seldom been of much utility (hence the strong demand for SSDs. Random IO matches the vast majority of workloads more appropriately).
Just caveats. This doesn't look like a terribly interesting option.
Good point, but their blog post does give a few possible uses: "Storage instances are ideal for data-intensive applications including Hadoop workloads, log processing and data warehousing, and parallel file systems to process and analyze large data sets in the AWS Cloud". If your code fits the pattern, a high-storage instance may fit well.
Of course, getting that much data into and out of the cloud is its own problem.
Even getting the data to that machine presents a problem that seems to undermine the value of it entirely: As you mentioned, the real value in this is linear processing of large sets of data, but the storage is ephemeral so your process has to be some variation of firing this instance up, copy TBs of data to the machine, and then do linear processing. Given that you have to get the data there, the value of the high aggregated gross-throughput seems secondary -- just stream process it, etc.
I'm having a tough time seeing where this type of instance fits.
That's why AWS Data Pipeline came along. http://aws.amazon.com/datapipeline/
Data Pipeline looks like a fine orchestration service, but it's not going to ingest 48 TB of data any faster than you can do it yourself. Which is probably not that fast.
maybe + http://aws.amazon.com/importexport/
> No one actually uses disks in such a fashion
Not sure what you mean... there are quite a few data storage & processing tools out there that use a log-structured on-disk storage that were designed to read/write sequentially. These could potentially take full advantage of these instances.
Many other solution requiring random IO is indeed a fact, but I still think there are systems that could benefit.
I mean that no one ever would configure 24 disks as RAID-0 -- the probability of a data-loss failure (which may not be a problem from the data-loss side, but is from the process continuity side) becomes incredibly high. Best case most people would arrange it as RAID-10, instantly dropping the performance by half.
If the use case is to only use the instance for several hours or days for some intense processing, then RAID-0 over 24 disks is probably just fine.