taken to its logical conclusion, tiered storage could turn Kafka into [...]
A message broker sitting in front of an RDBMS. I mean, if we're now basically 'tailing' streaming data and saving to another storage system might as well use RabbitMQ.
"tailing" is a pretty cool mechanism. An MQ is possibly too general.
For "tailing" you need to be able to resume, and that requires that you have a stable "offset" identifier, which in HTTP-speak would be: {URI, weak ETag, byte-range offset}, and then you can use conditional requests and local-part naming conventions to deal with things like:
- detection of rollover (which is where you can move older content to tiered storage),
- recovery from that (resumption), and
- detection of lost content (e.g., you resumed a tail a month later and [surprise!] you can't catch up and you need to recover in some other way.
In fact, I've written an HTTP server I call tailfhttpd that supports all of that and, if you GET w/ `Range: bytes=${offset}-` (for some offset) then the GET doesn't complete until the file is unlinked or renamed away. Poor-person's-Kafka. This is generally quite useful since you can use it to tail both structured and unstructured log files, as long as you only ever append to them.
Comments
A message broker sitting in front of an RDBMS. I mean, if we're now basically 'tailing' streaming data and saving to another storage system might as well use RabbitMQ.
"tailing" is a pretty cool mechanism. An MQ is possibly too general.
For "tailing" you need to be able to resume, and that requires that you have a stable "offset" identifier, which in HTTP-speak would be: {URI, weak ETag, byte-range offset}, and then you can use conditional requests and local-part naming conventions to deal with things like:
- detection of rollover (which is where you can move older content to tiered storage),
- recovery from that (resumption), and
- detection of lost content (e.g., you resumed a tail a month later and [surprise!] you can't catch up and you need to recover in some other way.
In fact, I've written an HTTP server I call tailfhttpd that supports all of that and, if you GET w/ `Range: bytes=${offset}-` (for some offset) then the GET doesn't complete until the file is unlinked or renamed away. Poor-person's-Kafka. This is generally quite useful since you can use it to tail both structured and unstructured log files, as long as you only ever append to them.
Kind of difficult if all you know is a cloud Kafka and have no idea that RabbitMQ exists or how to use it.
I went to a lunch and learn where a young engineer was demoing some Kafka-based solution that was spun up in AWS.
I asked about some of the functionality in comparison with RabbitMQ. Deer in the headlights.