Let's see, how do I put this gently; the data generated by high-throughput sequencing blows the living hell out of GWAS data. One exome requires reading 30-60x of what's read in a standard GWA. (Not to mention that most people get GWA data on their next-gen sequenced stuff anyways.) You're in the most data-intensive part of the field... congrats! As you probably know better than I, we're still in the age of targeted sequencing and exome sequencing. Whole genome is just beginning to dawn.
Where I work, filesystem I/O is the rate-limiting step for most of my (next-gen sequencing) experiments...
Yeah, we haven't even had to deal with data directly from the instrument yet. So far, we've been getting data from collaborators for analysis via terabyte usb drives (FedEx throughput can't be beat). For the actual analysis, we've found the same thing... disk IO is a limiting factor. Well, that and the 16GB human genome indexes in RAM. And we aren't on an Isilon system yet (probably won't be either).
However, we just got our own instrument, so this will definitely be an issue, but our University knows a thing or two about dealing with big data (http://kb.iu.edu/data/avvh.html).
I've only dealt with a few GWAS style datasets, and the next-gen stuff dwarfs the GWAS data in terms of size. But when looking for linkages between variations, we're still talking more time than the universe is old level of calculations for more than 3 combinations. Which is really scary, because like you said, all the genetics people are going to be using sequencing for most things from here on out, so its like you have complexity on top of complexity...
OK, now that I've thought about it, the easiest way around this probably is throwing money at hardware (more disks) or optimizing the processing. However, this is only in the case of a true disk I/O bottleneck. If you're optimizing correctly then the disks should be reading 8GB blocks directly into memory and the CPU should be spitting them right out again. At the very minimum you should be using an optimized file system with large pages enabled in your kernel.
I don't think I have enough low-level knowledge to answer that intelligently at the moment. What I can say is that we're hitting these problems despite being on Isilon drives. (I think that is orthogonal to your solution, but again am not all too familiar with the subject.)
Not really... these datasets start to saturate 10Gb network connections very easily, so it's a question of volume. With some instruments, you can generate 10-20 terabytes at a time.
Just DNA, on my end (human lipoprotein genetics). I know that the cancer folks have a lot of interest in RNA, however. What is your focus? (Edit: your profile pretty much explains it! )
Comments
Let's see, how do I put this gently; the data generated by high-throughput sequencing blows the living hell out of GWAS data. One exome requires reading 30-60x of what's read in a standard GWA. (Not to mention that most people get GWA data on their next-gen sequenced stuff anyways.) You're in the most data-intensive part of the field... congrats! As you probably know better than I, we're still in the age of targeted sequencing and exome sequencing. Whole genome is just beginning to dawn.
Where I work, filesystem I/O is the rate-limiting step for most of my (next-gen sequencing) experiments...
Yeah, we haven't even had to deal with data directly from the instrument yet. So far, we've been getting data from collaborators for analysis via terabyte usb drives (FedEx throughput can't be beat). For the actual analysis, we've found the same thing... disk IO is a limiting factor. Well, that and the 16GB human genome indexes in RAM. And we aren't on an Isilon system yet (probably won't be either).
However, we just got our own instrument, so this will definitely be an issue, but our University knows a thing or two about dealing with big data (http://kb.iu.edu/data/avvh.html).
I've only dealt with a few GWAS style datasets, and the next-gen stuff dwarfs the GWAS data in terms of size. But when looking for linkages between variations, we're still talking more time than the universe is old level of calculations for more than 3 combinations. Which is really scary, because like you said, all the genetics people are going to be using sequencing for most things from here on out, so its like you have complexity on top of complexity...
Could non-blocking filesystem I/O (a la node.js: http://blog.kodekabuki.com/post/267934877/how-node-js-expose... ) improve that?
Final edit:
OK, now that I've thought about it, the easiest way around this probably is throwing money at hardware (more disks) or optimizing the processing. However, this is only in the case of a true disk I/O bottleneck. If you're optimizing correctly then the disks should be reading 8GB blocks directly into memory and the CPU should be spitting them right out again. At the very minimum you should be using an optimized file system with large pages enabled in your kernel.
I don't think I have enough low-level knowledge to answer that intelligently at the moment. What I can say is that we're hitting these problems despite being on Isilon drives. (I think that is orthogonal to your solution, but again am not all too familiar with the subject.)
Not really... these datasets start to saturate 10Gb network connections very easily, so it's a question of volume. With some instruments, you can generate 10-20 terabytes at a time.
wow, what do you do?
One of my hats is being an analyst dealing with exome (all exons in the genome) sequencing for my lab. We do a lot of exomes.
That would be great at a party:
"What do you do for a living?"
"Exomes."
-spits drink out-
Hah! If that would go over well at a party of yours, you'll need to introduce me to your friends :)
We are dealing with transcriptomes. Do you do targeted sequencing of exomes (DNA), or RNA?
Just DNA, on my end (human lipoprotein genetics). I know that the cancer folks have a lot of interest in RNA, however. What is your focus? (Edit: your profile pretty much explains it! )