It's practically the definition of data warehousing that its whole purpose in life is to deal with everyone else's bullshit. If you want to combine data from different sources, you have to retroactive fix all the mistakes that the data owners made that don't cause issues for them but do cause issues for you.
Story of my life in data science, right there. We work hard to build a culture of shared data ownership, where the data producers have ownership and responsibility of the data they generate, rather than just lobbing garbage over the wall for us to deal with. But it'll always be hard-- as the ultimate the end users of the data, data science/analytics/business ops are always going to care most about its quality.
Comments
It's practically the definition of data warehousing that its whole purpose in life is to deal with everyone else's bullshit. If you want to combine data from different sources, you have to retroactive fix all the mistakes that the data owners made that don't cause issues for them but do cause issues for you.
Story of my life in data science, right there. We work hard to build a culture of shared data ownership, where the data producers have ownership and responsibility of the data they generate, rather than just lobbing garbage over the wall for us to deal with. But it'll always be hard-- as the ultimate the end users of the data, data science/analytics/business ops are always going to care most about its quality.