I am curious, what are my "unsafe assumptions"? I would love more detailed feedback.
ETL is not the biggest problem, but it is very time consuming, tedious, and costly because the tools overall suck. That is why I started Datalanche although to be fair we so far only solve part of the problem (storage, query, sharing).
Anyway the OP wants to learn how to do big data and make money. Well even if you are doing the cool stuff such as analytics, you still need to know ETL basics. I offered to give some experience doing that.
Not your unsafe assumptions but in stats almost every blind alley and wrong turn is an assumption that bit back. Assumed that the data we have is a representative sample of all purchasers etc etc
I have no real knowledge of the big data market so am happy to defer. I assume you are taking data sets covering g them same thing in hundreds or thousands of hospitals nationwide and outputting nicely mapped nationwide data?
Please do give as much info on the market as you can - always interested in hearing from on the ground "troops"
The CDC has made available two datasets which are of de-identified medical records going back 20 years which are updated yearly. Unforunately it is extremely difficult to find the data. In fact, I went to the correct web page and missed it. One of my colleagues found it instead. The data is in a custom text format which needs a custom parser, and the format changes slightly every year. Then codes and abbreviations are used everywhere in the data set which need to be determined and converted. Then you have the usual misspellings, etc that are common in government data sets.
It is truly a pain to use these CDC data sets even though the data itself is great.
I don't really know. I have an educated guess for the custom format though. They started collecting data before XML, JSON, etc were widely used and/or existed which explains the custom text format. Since their systems already handle their custom format, there is no real incentive to change it.
I see. Yes I would agree that unsafe assumptions are a big problem when doing statistics regardless of data size, and probably one of the biggest if not the biggest.
My long-term hope is that people will start sharing data they have already formatted, cleaned, etc. This will save others time. So one of the goals of Datalanche is to make the sharing mechanic dead simple. Doesn't mean sharing will happen, but hopefully it makes it easier for those who do want to share.
Therefore we are populating our own website with public data starting with healthcare. I have personal experience with healthcare data analytics and have some sense of which data sets are useful.
Comments
I am curious, what are my "unsafe assumptions"? I would love more detailed feedback.
ETL is not the biggest problem, but it is very time consuming, tedious, and costly because the tools overall suck. That is why I started Datalanche although to be fair we so far only solve part of the problem (storage, query, sharing).
Anyway the OP wants to learn how to do big data and make money. Well even if you are doing the cool stuff such as analytics, you still need to know ETL basics. I offered to give some experience doing that.
Not your unsafe assumptions but in stats almost every blind alley and wrong turn is an assumption that bit back. Assumed that the data we have is a representative sample of all purchasers etc etc
I have no real knowledge of the big data market so am happy to defer. I assume you are taking data sets covering g them same thing in hundreds or thousands of hospitals nationwide and outputting nicely mapped nationwide data?
Please do give as much info on the market as you can - always interested in hearing from on the ground "troops"
Just to give you a more concrete example.
The CDC has made available two datasets which are of de-identified medical records going back 20 years which are updated yearly. Unforunately it is extremely difficult to find the data. In fact, I went to the correct web page and missed it. One of my colleagues found it instead. The data is in a custom text format which needs a custom parser, and the format changes slightly every year. Then codes and abbreviations are used everywhere in the data set which need to be determined and converted. Then you have the usual misspellings, etc that are common in government data sets.
It is truly a pain to use these CDC data sets even though the data itself is great.
Can you answer why it's so bad?
I can guess at a lot of answers but which particular strain of corporate pathology CDC suffers from
I don't really know. I have an educated guess for the custom format though. They started collecting data before XML, JSON, etc were widely used and/or existed which explains the custom text format. Since their systems already handle their custom format, there is no real incentive to change it.
I see. Yes I would agree that unsafe assumptions are a big problem when doing statistics regardless of data size, and probably one of the biggest if not the biggest.
My long-term hope is that people will start sharing data they have already formatted, cleaned, etc. This will save others time. So one of the goals of Datalanche is to make the sharing mechanic dead simple. Doesn't mean sharing will happen, but hopefully it makes it easier for those who do want to share.
Therefore we are populating our own website with public data starting with healthcare. I have personal experience with healthcare data analytics and have some sense of which data sets are useful.