Kafka topics compacted to some sort of disk storage? It really depends on what you need to do with it. All of our kafka topics have a TTL on messages that's less than or equal to 7 days. If the data is critical, then that topic has a archiver job that writes the messages to files in HDFS.
Postgres, Spanner, etc. come to mind as options if you need to store critical append-only data and also have it be queryable in real-time.
postgres is a bit different use case, isn't? kafka is mainly about scalability, it has nice partitioning out of the box. and i would not say that it's not reliable, because you always replicate data with it
Comments
Kafka topics compacted to some sort of disk storage? It really depends on what you need to do with it. All of our kafka topics have a TTL on messages that's less than or equal to 7 days. If the data is critical, then that topic has a archiver job that writes the messages to files in HDFS.
Postgres, Spanner, etc. come to mind as options if you need to store critical append-only data and also have it be queryable in real-time.
postgres is a bit different use case, isn't? kafka is mainly about scalability, it has nice partitioning out of the box. and i would not say that it's not reliable, because you always replicate data with it