Comment on Scaling request logging with ClickHouse, Kafka, and VectorComments−est10mocan you just buffer some writes in Vector and eliminate Kafka?I setup some Vector to buffer ElasticSearch writes years ago, also for logs, it ran so well without any problems that I almost fogot about it.−anticodon10moOr vice versa: make ClickHouse ingest batches directly from Kafka. Messages are already buffered in Kafka, I don't get why Vector is necessary here.−ch202610motbh the only thing they needed was a correct schema that didn’t constantly spawn new parts and async_insert enabled.−est10mohttps://clickhouse.com/docs/knowledgebase/kafka-to-clickhous...For anyone if curious.
Comments
can you just buffer some writes in Vector and eliminate Kafka?
I setup some Vector to buffer ElasticSearch writes years ago, also for logs, it ran so well without any problems that I almost fogot about it.
Or vice versa: make ClickHouse ingest batches directly from Kafka. Messages are already buffered in Kafka, I don't get why Vector is necessary here.
tbh the only thing they needed was a correct schema that didn’t constantly spawn new parts and async_insert enabled.
https://clickhouse.com/docs/knowledgebase/kafka-to-clickhous...
For anyone if curious.