I found in integration/warehousing that if a source system “suddenly” has a new column — it’s best if we can automatically just bring it in. We would tend to do this in a way that it can’t break the warehouse, doesn’t affect people downstream of us. We can then choose to make it available to others, or not — but the moment that data is available we start hoovering it in.
That’s the logical thing in the context of this domain. In a broader data domain there are other considerations, not always logical, so if that ends up configurable it’s a feature that broadens application scope.
Comments
I found in integration/warehousing that if a source system “suddenly” has a new column — it’s best if we can automatically just bring it in. We would tend to do this in a way that it can’t break the warehouse, doesn’t affect people downstream of us. We can then choose to make it available to others, or not — but the moment that data is available we start hoovering it in.
That’s the logical thing in the context of this domain. In a broader data domain there are other considerations, not always logical, so if that ends up configurable it’s a feature that broadens application scope.