forgetting a small slice of data from a large set a complicated and nearly impossible task to accomplish.
We use Kafka as our storage for almost everything, and we managed to solve this by encrypting all user data that is relevant to GDPR and trowing away the key when asked for a removal.
if a user asks to be forgotten, we commit a empty privacy key for this user and compress the privacykeys topic and all is done, no service will be able to decrypt it anymore.
So far it has been a good solution and it was easy to implement on all our services.
This is a great idea. If you couple this practice with storing user data encryption keys with additional layers of security, you’ll decrease your susceptibility of someone being able to extract all data if they get access to your kafka. Spotify talks about this here: http://labs.spotify.com/2018/09/18/scalable-user-privacy/
This practice - cryptoshredding - works well with two caveats.
First, it requires some policing of Kafka use. It's easy for developers to slip up and some PII to spill into the append-only data systems.
Second, your developers will have to handle for what happens when the key is deleted. The happy-path of fetching data, fetching key, and applying will fail quite hard the first time the rare event of a key deletion comes around.
Comments
We use Kafka as our storage for almost everything, and we managed to solve this by encrypting all user data that is relevant to GDPR and trowing away the key when asked for a removal.
if a user asks to be forgotten, we commit a empty privacy key for this user and compress the privacykeys topic and all is done, no service will be able to decrypt it anymore.
So far it has been a good solution and it was easy to implement on all our services.
This is a great idea. If you couple this practice with storing user data encryption keys with additional layers of security, you’ll decrease your susceptibility of someone being able to extract all data if they get access to your kafka. Spotify talks about this here: http://labs.spotify.com/2018/09/18/scalable-user-privacy/
This practice - cryptoshredding - works well with two caveats.
First, it requires some policing of Kafka use. It's easy for developers to slip up and some PII to spill into the append-only data systems.
Second, your developers will have to handle for what happens when the key is deleted. The happy-path of fetching data, fetching key, and applying will fail quite hard the first time the rare event of a key deletion comes around.