Ask HN: What tool do you use to sync MongoDB with Elasticsearch?
Currently I am performing full text search on MongoDB but I would love to use Elasticsearch for the same. My primary data source is MongoDB. Right now I am not able find a suitable/reliable sync tool for MongoDB->Elasticsearch.
There used to be a tool called elasticsearch-river-mongodb [1] which is deprecated now.
Also, there is no official logstash input plugin for MongoDB
[1] https://github.com/richardwilly98/elasticsearch-river-mongodb
What tool do you use to reliably sync MongoDB collections with Elasticsearch?
Comments
Take a look at Kafka. When your backend pushes data to MongoDB, have it produce a Kafka message that contains serialized Elasticsearch JSON documents and then use either a bespoke consumer, or kafka-connect-elasticsearch, or logstash-input-kafka to consume those documents and bulk-insert them into Elasticsearch.
Bonus points: it will let you have a standby Elasticsearch cluster in another region you could failover to when needed, perform Elasticsearch cluster upgrades/maintenance without losing data, and more.
Depending on your use case this could be an overkill so YMMV.
I could definitely do that. But my original intention was to use a standalone application which can get the job done. For example when the data is updated/deleted instead of insertion I need to write my own logic.
Any way great suggestion. Will definitely keep in consideration.
https://github.com/indatawetrust/mongo-to-elastic