Hi HN, we created ABC import as a convenient way to sync data source to an Elasticsearch index. It does three things really well imho:
1. A small footprint process / docker container that can index or sync your data source with ES that is operationally simple vs relying on application layer logic,
2. Supports on-the-fly transformations with Javascript, as well as configuration of mappings (so if you want to set a specific analyzer on your Text fields, or set type mappings),
3. Works with a wide variety of sources - Postgres, MySQL, SQL Server, MongoDB, JSON, CSV, Elasticsearch and more coming soon.
1. and 2. It syncs deltas if you are either using Postgres (9.4 or above) or MongoDB. For other sources like MySQL or MSSQL, it needs to import the whole table each time. The tailing capability is DB dependent, for instance, in MongoDB's case - we use the oplog to do this.
Thanks! We haven't tested it at scale, but this is where we would ideally want it to be at v1. If you would like to take it for a spin, would appreciate any feedback on scaling limits.
Comments
Hi HN, we created ABC import as a convenient way to sync data source to an Elasticsearch index. It does three things really well imho:
1. A small footprint process / docker container that can index or sync your data source with ES that is operationally simple vs relying on application layer logic,
2. Supports on-the-fly transformations with Javascript, as well as configuration of mappings (so if you want to set a specific analyzer on your Text fields, or set type mappings),
3. Works with a wide variety of sources - Postgres, MySQL, SQL Server, MongoDB, JSON, CSV, Elasticsearch and more coming soon.
A couple questions with regards to the SQL -> ES:
1. Does it sync deltas or do you have to import the whole table each time?
2. Does it listen for changes to a table or does it require a manual invocation?
3. Can you explain more about the algorithm/implementation?
1. and 2. It syncs deltas if you are either using Postgres (9.4 or above) or MongoDB. For other sources like MySQL or MSSQL, it needs to import the whole table each time. The tailing capability is DB dependent, for instance, in MongoDB's case - we use the oplog to do this.
The way it works is after the initial import if you are using the `--tail` option, it listens to further changes via oplog. More on setting oplog access over here - https://github.com/appbaseio/abc/blob/dev/docs/importer/adap...
For Postgres, similarly a replication role and a replication slot needs to be created for decoding the write ahead log. More on that here - https://github.com/appbaseio/abc/blob/dev/docs/importer/adap....
This seems like a good primer on replication slots - https://postgresqlspace.wordpress.com/2015/06/18/replication....
Looks like a cool project.
Does the real-time sync work for Postgres instances that are tens of GB and thousands of tables across schemas?
Thanks! We haven't tested it at scale, but this is where we would ideally want it to be at v1. If you would like to take it for a spin, would appreciate any feedback on scaling limits.