Not the original comment, but I have run into a specific set of regulatory scenarios where the immutability of any event log is a big problem.
The data retention policies for users who leave the platform is set to 30 days by GDPR, which requires the data to be deleted and expunged by the storage system in ways in which normal users cannot recover - or in another way, the data needs to "offline".
This is not actually that "all data older than 30 days is thrown away", but that "all data for users who have deleted their accounts need to be deleted from the start of their account creation, 30 days after they say forget-me".
Kafka (or Pulsar or Pravega) or any of the other immutable commit log implementations make forgetting a small slice of data from a large set a complicated and nearly impossible task to accomplish.
You can accomplish some part of this with log compaction assuming you have a definite primary key for all updates (i.e the log needs to be partitioned on a key to do compaction along that key). If there's a way I could declare a primary key as a device+event_timestamp+metric, but delete by a user_id column, let me know and I'll be happy to find out how to do it.
However in its original form, Kafka is still very useful.
Being able to hold data in Kafka in those periods of time is extremely valuable and naturally lets the system lose part of its state with the ability to replay itself back into the same state from a saved checkpoint.
If you store 7+ days of Kafka data and flush the newly arrived data-set into a persistent, but mutable columnar store every day & maintain the partition/offsets on commit, then you can recover from a complete loss of the mutable store's in-memory data by replaying the log from where you left off.
The row-major nature of its storage still hurts though if you plan to do all your analysis off it directly, because you'll burn through the disk bandwidth for no good reason.
forgetting a small slice of data from a large set a complicated and nearly impossible task to accomplish.
We use Kafka as our storage for almost everything, and we managed to solve this by encrypting all user data that is relevant to GDPR and trowing away the key when asked for a removal.
if a user asks to be forgotten, we commit a empty privacy key for this user and compress the privacykeys topic and all is done, no service will be able to decrypt it anymore.
So far it has been a good solution and it was easy to implement on all our services.
This is a great idea. If you couple this practice with storing user data encryption keys with additional layers of security, you’ll decrease your susceptibility of someone being able to extract all data if they get access to your kafka. Spotify talks about this here: http://labs.spotify.com/2018/09/18/scalable-user-privacy/
This practice - cryptoshredding - works well with two caveats.
First, it requires some policing of Kafka use. It's easy for developers to slip up and some PII to spill into the append-only data systems.
Second, your developers will have to handle for what happens when the key is deleted. The happy-path of fetching data, fetching key, and applying will fail quite hard the first time the rare event of a key deletion comes around.
If you consider the log streams as backups then GDPR doesn't apply.
According to France's GDPR supervisory authority, CNIL, organisations don't have to delete backups when complying with the right to erasure. Nonetheless, they must clearly explain to the data subject that backups will be kept for a specified length of time (outlined in your retention policy).
Comments
This is quite an unsubstantiated claim in response to a detailed article. Care to elaborate?
Not the original comment, but I have run into a specific set of regulatory scenarios where the immutability of any event log is a big problem.
The data retention policies for users who leave the platform is set to 30 days by GDPR, which requires the data to be deleted and expunged by the storage system in ways in which normal users cannot recover - or in another way, the data needs to "offline".
This is not actually that "all data older than 30 days is thrown away", but that "all data for users who have deleted their accounts need to be deleted from the start of their account creation, 30 days after they say forget-me".
Kafka (or Pulsar or Pravega) or any of the other immutable commit log implementations make forgetting a small slice of data from a large set a complicated and nearly impossible task to accomplish.
You can accomplish some part of this with log compaction assuming you have a definite primary key for all updates (i.e the log needs to be partitioned on a key to do compaction along that key). If there's a way I could declare a primary key as a device+event_timestamp+metric, but delete by a user_id column, let me know and I'll be happy to find out how to do it.
However in its original form, Kafka is still very useful.
Being able to hold data in Kafka in those periods of time is extremely valuable and naturally lets the system lose part of its state with the ability to replay itself back into the same state from a saved checkpoint.
If you store 7+ days of Kafka data and flush the newly arrived data-set into a persistent, but mutable columnar store every day & maintain the partition/offsets on commit, then you can recover from a complete loss of the mutable store's in-memory data by replaying the log from where you left off.
The row-major nature of its storage still hurts though if you plan to do all your analysis off it directly, because you'll burn through the disk bandwidth for no good reason.
We use Kafka as our storage for almost everything, and we managed to solve this by encrypting all user data that is relevant to GDPR and trowing away the key when asked for a removal.
if a user asks to be forgotten, we commit a empty privacy key for this user and compress the privacykeys topic and all is done, no service will be able to decrypt it anymore.
So far it has been a good solution and it was easy to implement on all our services.
This is a great idea. If you couple this practice with storing user data encryption keys with additional layers of security, you’ll decrease your susceptibility of someone being able to extract all data if they get access to your kafka. Spotify talks about this here: http://labs.spotify.com/2018/09/18/scalable-user-privacy/
This practice - cryptoshredding - works well with two caveats.
First, it requires some policing of Kafka use. It's easy for developers to slip up and some PII to spill into the append-only data systems.
Second, your developers will have to handle for what happens when the key is deleted. The happy-path of fetching data, fetching key, and applying will fail quite hard the first time the rare event of a key deletion comes around.
If you consider the log streams as backups then GDPR doesn't apply.
According to France's GDPR supervisory authority, CNIL, organisations don't have to delete backups when complying with the right to erasure. Nonetheless, they must clearly explain to the data subject that backups will be kept for a specified length of time (outlined in your retention policy).