The CDC has made available two datasets which are of de-identified medical records going back 20 years which are updated yearly. Unforunately it is extremely difficult to find the data. In fact, I went to the correct web page and missed it. One of my colleagues found it instead. The data is in a custom text format which needs a custom parser, and the format changes slightly every year. Then codes and abbreviations are used everywhere in the data set which need to be determined and converted. Then you have the usual misspellings, etc that are common in government data sets.
It is truly a pain to use these CDC data sets even though the data itself is great.
I don't really know. I have an educated guess for the custom format though. They started collecting data before XML, JSON, etc were widely used and/or existed which explains the custom text format. Since their systems already handle their custom format, there is no real incentive to change it.
Comments
Just to give you a more concrete example.
The CDC has made available two datasets which are of de-identified medical records going back 20 years which are updated yearly. Unforunately it is extremely difficult to find the data. In fact, I went to the correct web page and missed it. One of my colleagues found it instead. The data is in a custom text format which needs a custom parser, and the format changes slightly every year. Then codes and abbreviations are used everywhere in the data set which need to be determined and converted. Then you have the usual misspellings, etc that are common in government data sets.
It is truly a pain to use these CDC data sets even though the data itself is great.
Can you answer why it's so bad?
I can guess at a lot of answers but which particular strain of corporate pathology CDC suffers from
I don't really know. I have an educated guess for the custom format though. They started collecting data before XML, JSON, etc were widely used and/or existed which explains the custom text format. Since their systems already handle their custom format, there is no real incentive to change it.