This practice - cryptoshredding - works well with two caveats.
First, it requires some policing of Kafka use. It's easy for developers to slip up and some PII to spill into the append-only data systems.
Second, your developers will have to handle for what happens when the key is deleted. The happy-path of fetching data, fetching key, and applying will fail quite hard the first time the rare event of a key deletion comes around.
Comments
This practice - cryptoshredding - works well with two caveats.
First, it requires some policing of Kafka use. It's easy for developers to slip up and some PII to spill into the append-only data systems.
Second, your developers will have to handle for what happens when the key is deleted. The happy-path of fetching data, fetching key, and applying will fail quite hard the first time the rare event of a key deletion comes around.