Oh wow ! I was just trying to articulate exactly this kind of approach and looking for it. I would love to see a coherent approach to data errors and this seems like a step in the right direction.
One question - the blog post covers basically debugging the ingestion of data part. My quite usual issue with older data is that at some point, you discover an issue with it (say it's slightly false, but not too much) - so you want to somehow let users know, or allow to select only the data without the issue (but still let them know how much of it they miss) - is this framework helpful in this situation ?
Yes, great point! The idea is that you can fix up a problem with data in place, while you're updating your ingest pipeline to handle whatever is causing the problem. You can do a transform on the errors into clean data, delete the errors, and commit the changes atomically. In the meantime, queries and searches can still run on the data that isn't problematic and even if there are errors inside of a hierarchical value, queries can be run on the portions of the value that are clean and intact while the errors are being addressed.
Comments
Oh wow ! I was just trying to articulate exactly this kind of approach and looking for it. I would love to see a coherent approach to data errors and this seems like a step in the right direction.
One question - the blog post covers basically debugging the ingestion of data part. My quite usual issue with older data is that at some point, you discover an issue with it (say it's slightly false, but not too much) - so you want to somehow let users know, or allow to select only the data without the issue (but still let them know how much of it they miss) - is this framework helpful in this situation ?
Yes, great point! The idea is that you can fix up a problem with data in place, while you're updating your ingest pipeline to handle whatever is causing the problem. You can do a transform on the errors into clean data, delete the errors, and commit the changes atomically. In the meantime, queries and searches can still run on the data that isn't problematic and even if there are errors inside of a hierarchical value, queries can be run on the portions of the value that are clean and intact while the errors are being addressed.