I think what the article greatly skimps over is data migrations: what do you do if you need to change the format of your data? If you retain logs in Kafka indefinitely as the source of truth for your data, then if you need to migrate materialized data to a new format, you'll also need to either 1) support all the previous forms of materialized data so operations from the log are guaranteed to be safely replayable on it, or 2) don't do that and keep one form of materialized data and hope you have enough test coverage to make sure some unexpectedly old data doesn't silently corrupt your materialized data.
Event sourcing is useful, but using it as a source of truth data store in itself instead of e.g. an occasional journalling mechanism seems pretty fraught.
You use AVRO and the confluent schema store so that you can do compatible schema upgrades (automatically upgrading messages when reading). If you need to do a breaking change, you can detect the avro schema from the message and have code in the reader to convert one to the other. It’s not that much of a problem in practice.
This is a problem we've actually encountered in practice: you need to preserve the conversion code for as long as the history exists, which is permanent technical debt. If you get your domain model wrong at the start, it will haunt you forever.
You can truncate the history by reading events from the starting point to the junction, then creating a new snapshot that will serve as a new starting point, and finally deleting the previous history segment.
This is definitely a decision which requires a long term support, as any solution regarding storage. This is not a technical debt unless you see it as a temporary solution, waiting for a better time to replace it.
You highlight the "need to preserve the conversion code for as long as the history exists". This is a point so decisive that I even wonder if Kafka should provide some support to keep that relationship.
If you write "read it to the latest version" code for every version, you need to write that for every past version - potentially a high burden. Writing "convert it to the next version" instead is much less work for humans, and probably usually the best call, but less efficient when you actually need to read that old data. There are hybrid approaches to consider, too.
Wherever you are on that spectrum, there is code. Exercising it may slow down your test suite. It might need to be touched when you update lint rules (at least to add annotations to shut off the new rules).
Code nearly always represents some amount of debt. Code to deal with history, more so.
We have a slight disagreement on what it means to accrue technical debt.
When a schema changes for an all, someone has to migrate it and the data, unless the design is that all incompatible data is dropped. Either that’s the codebase itself or the dev team is punting to the user. Punting to the user just shifts the burden around.
If the R&D team shipped a shitty / incomplete schema to get the software out the door that they need to change later, then yes, that is technical debt - something they’ll eventually need to repay.
If requirements evolve over time and thus the schema needs to, that is not necessarily technical debt in the usual sense, which usually implies a temporary technical compromise for the purposes of expediency / getting something out the door.
I suppose I agree there is a trade off here and a penalty - after many years the migrations get slow to apply etc, and you could say that checkpointing the schema every few versions and preventing upgrades from any prior point is a way of cleaning up the “migration debt”.
But people like to suggest that there’s some other way , ie. the OP saying “ If you get your domain model wrong at the start, it will haunt you forever.”.... I have never seen any system get the domain model perfectly right at the start!
I personally don't understand avro, are there tools to write the schemas easier? The json format is very difficult to read and it just seems so clunky compared to protobuf.
> 1) support all the previous forms of materialized data so operations from the log are guaranteed to be safely replayable on it
Yes, I believe with event sourcing you typically do exactly that. The very point of it is to use the event log as the source of truth, not just an “occasional journaling mechanism.”
I've worked on maintaining systems like that. IME, it's classic technical debt: You can get up and running much more quickly and cheaply, but in the long run it can become quite expensive. So, like any good technical debt, it's fine, even desirable, in the short run, but less fine if you end up carrying it indefinitely.
Fortunately, Kafka's already thought about this, so, when the time comes, it should be pretty easy to migrate to an architecture where you're no longer using it as your long-term source of truth.
People go out there way to make events the source of truth in non-kafka systems. With a lot of effort involved. It isn't just people being lazy with Kafka. Event sourcing existed before with traditional databases.
I came across it in the .net world with cqrs + event sourcing combo long before Kafka became popular. People put a lot effort in to move away traditional current state storage. Often used in industries that have a lot of regulation.
I think what GP meant is: what if one of your "indexes" is a SQL database, and over time the shape that database needs to be changes?
In that situation, how do you handle migrations on that database, or building a new copy of it from scratch via the log? Your log will have historical data in a different shape than the schema expects, so you'll end up in an uncomfortable "replay a little, migrate, repeat" situation when reading historical data into the index for any reason.
Comments
I think what the article greatly skimps over is data migrations: what do you do if you need to change the format of your data? If you retain logs in Kafka indefinitely as the source of truth for your data, then if you need to migrate materialized data to a new format, you'll also need to either 1) support all the previous forms of materialized data so operations from the log are guaranteed to be safely replayable on it, or 2) don't do that and keep one form of materialized data and hope you have enough test coverage to make sure some unexpectedly old data doesn't silently corrupt your materialized data.
Event sourcing is useful, but using it as a source of truth data store in itself instead of e.g. an occasional journalling mechanism seems pretty fraught.
You use AVRO and the confluent schema store so that you can do compatible schema upgrades (automatically upgrading messages when reading). If you need to do a breaking change, you can detect the avro schema from the message and have code in the reader to convert one to the other. It’s not that much of a problem in practice.
This is a problem we've actually encountered in practice: you need to preserve the conversion code for as long as the history exists, which is permanent technical debt. If you get your domain model wrong at the start, it will haunt you forever.
You can truncate the history by reading events from the starting point to the junction, then creating a new snapshot that will serve as a new starting point, and finally deleting the previous history segment.
This is definitely a decision which requires a long term support, as any solution regarding storage. This is not a technical debt unless you see it as a temporary solution, waiting for a better time to replace it.
You highlight the "need to preserve the conversion code for as long as the history exists". This is a point so decisive that I even wonder if Kafka should provide some support to keep that relationship.
This is no different from evolving an RDBMS with ActiveRecord or Flyway. It’s not technical debt (there is nothing to repay or maintain).
If you write "read it to the latest version" code for every version, you need to write that for every past version - potentially a high burden. Writing "convert it to the next version" instead is much less work for humans, and probably usually the best call, but less efficient when you actually need to read that old data. There are hybrid approaches to consider, too.
Wherever you are on that spectrum, there is code. Exercising it may slow down your test suite. It might need to be touched when you update lint rules (at least to add annotations to shut off the new rules).
Code nearly always represents some amount of debt. Code to deal with history, more so.
We have a slight disagreement on what it means to accrue technical debt.
When a schema changes for an all, someone has to migrate it and the data, unless the design is that all incompatible data is dropped. Either that’s the codebase itself or the dev team is punting to the user. Punting to the user just shifts the burden around.
If the R&D team shipped a shitty / incomplete schema to get the software out the door that they need to change later, then yes, that is technical debt - something they’ll eventually need to repay.
If requirements evolve over time and thus the schema needs to, that is not necessarily technical debt in the usual sense, which usually implies a temporary technical compromise for the purposes of expediency / getting something out the door.
I suppose I agree there is a trade off here and a penalty - after many years the migrations get slow to apply etc, and you could say that checkpointing the schema every few versions and preventing upgrades from any prior point is a way of cleaning up the “migration debt”.
But people like to suggest that there’s some other way , ie. the OP saying “ If you get your domain model wrong at the start, it will haunt you forever.”.... I have never seen any system get the domain model perfectly right at the start!
That may be. I was mostly trying to address the claim that "[i]t's not technical debt [because] there is nothing to [...] maintain."
Financially speaking, it's more like an annuity than a credit card.
So, like a set of traditional database migrations?
I personally don't understand avro, are there tools to write the schemas easier? The json format is very difficult to read and it just seems so clunky compared to protobuf.
> 1) support all the previous forms of materialized data so operations from the log are guaranteed to be safely replayable on it
Yes, I believe with event sourcing you typically do exactly that. The very point of it is to use the event log as the source of truth, not just an “occasional journaling mechanism.”
I've worked on maintaining systems like that. IME, it's classic technical debt: You can get up and running much more quickly and cheaply, but in the long run it can become quite expensive. So, like any good technical debt, it's fine, even desirable, in the short run, but less fine if you end up carrying it indefinitely.
Fortunately, Kafka's already thought about this, so, when the time comes, it should be pretty easy to migrate to an architecture where you're no longer using it as your long-term source of truth.
People go out there way to make events the source of truth in non-kafka systems. With a lot of effort involved. It isn't just people being lazy with Kafka. Event sourcing existed before with traditional databases.
I came across it in the .net world with cqrs + event sourcing combo long before Kafka became popular. People put a lot effort in to move away traditional current state storage. Often used in industries that have a lot of regulation.
Event sourcing may be many things but I don't think "quick and cheap" is one of them. If you want quick and cheap do a (distributed) monolith.
The entire point of event sourcing is that it is the source of truth. The sourcing in event sourcing.
Otherwise it's not different from a audit log.
I think what GP meant is: what if one of your "indexes" is a SQL database, and over time the shape that database needs to be changes?
In that situation, how do you handle migrations on that database, or building a new copy of it from scratch via the log? Your log will have historical data in a different shape than the schema expects, so you'll end up in an uncomfortable "replay a little, migrate, repeat" situation when reading historical data into the index for any reason.
Normally you version your events, uplift or convert a stream to a updated version.