A lot of this isn't related to data processing tools at all, but is a sort of downstream affect of the predominant "bugs are cheap" mentality of today.
The less guarantees of correctness on your daily/weekly/whatever releases, the messier your downstream data is gonna be. Monday's data is partially missing due to a bug in the client; Tuesday's data is weird/nonrepresentative because of a server bug that caused 5% of sessions to get disconnected; Wednesday's data is good; Thursday's data is good but was a release day and the feature changed so it means different stuff...
Comments
A lot of this isn't related to data processing tools at all, but is a sort of downstream affect of the predominant "bugs are cheap" mentality of today.
The less guarantees of correctness on your daily/weekly/whatever releases, the messier your downstream data is gonna be. Monday's data is partially missing due to a bug in the client; Tuesday's data is weird/nonrepresentative because of a server bug that caused 5% of sessions to get disconnected; Wednesday's data is good; Thursday's data is good but was a release day and the feature changed so it means different stuff...