So many people in this discussion take sides. The only thing I see, is divergence. APIs will now evolve separately, and compatibility will break soon. I think this competition and diversification will be good for innovation in search. But it’s also going to be confusing for a while. If you use Elasticsearch in Amazon, you better decide very quickly whether you want to hitch your wagon to the OpenSearch future. Migrating is going to be a pain in a year or two.
This didn't happen with MySQL/MariaDB split -- they are still mostly compatible.
Granted, it looks like Elastic is trying to break compatibility pretty hard (see the new version checks in clients for example) but hopefully Amazon will release universal connectors, it certainly in the Amazon's interests.
Good point about MySQL/MariaDB. I think this is different though because search engines are at a big pivot point to include approximate-nearest-neighbor dense vector search (it has before been only sparse vector search for Lucene based platforms).
Specifically, this feature https://github.com/elastic/elasticsearch/issues/42326#issuec... will be a big change for Elasticsearch. OpenSearch might try to mimic the API, but implementation details here will matter a lot, since this type of search is really picky when it comes to performance/recall balance. OpenDistro has already been working on their own version: https://opendistro.github.io/for-elasticsearch/features/knn.... ...so will they switch their API? Perhaps - but the results are going to be very different.
It is pretty clear to me that Elastic is planning to build their ANN features differently than OpenDistro's k-NN implementation, or other plugins modules that extend Easticsearch in similar ways. They now will build on the Apache Lucene capabilities that were collaboratively built "upstream" by a number of individuals, some that work for Amazon and some that work for Elastic.
From the linked issue, it seemed that they were originally planning to develop this as a proprietary feature of Elasticsearch, without contributing the functionality to Apache Lucene, but then changed direction when the Apache Lucene developers (some of which are currently employed to do such work by Amazon) started to build its approximate nearest neighbor (ANN) vector search capabilities. [1]
It's great to see folks that work for Elastic collaborating and building on what is in Apache Lucene to extend the utility of ANN with Hierarchical Navigable Small World Graphs (HNSW) [2]! From this, I think it should be possible to implement an Open Source version of the functionality with a compatible API, if that is something that OpenSearch users seek.
Why would most companies need to migrate if they're happy with the features in a past version? Why not stick with it (assuming there are no major security flaws?)
Well, how far is Amazon going to take OpenSearch? Elastic is all in on innovating and building their flagship product forever with significant dedication. AWS just needs something good enough as an offering to go with the rest of their OK cloud services.
I think most people can see both sides of this conflict. But, as you yourself pointed out at the end of your comment, finding a side to pick quickly is important.
Comments
So many people in this discussion take sides. The only thing I see, is divergence. APIs will now evolve separately, and compatibility will break soon. I think this competition and diversification will be good for innovation in search. But it’s also going to be confusing for a while. If you use Elasticsearch in Amazon, you better decide very quickly whether you want to hitch your wagon to the OpenSearch future. Migrating is going to be a pain in a year or two.
This didn't happen with MySQL/MariaDB split -- they are still mostly compatible.
Granted, it looks like Elastic is trying to break compatibility pretty hard (see the new version checks in clients for example) but hopefully Amazon will release universal connectors, it certainly in the Amazon's interests.
Good point about MySQL/MariaDB. I think this is different though because search engines are at a big pivot point to include approximate-nearest-neighbor dense vector search (it has before been only sparse vector search for Lucene based platforms).
Specifically, this feature https://github.com/elastic/elasticsearch/issues/42326#issuec... will be a big change for Elasticsearch. OpenSearch might try to mimic the API, but implementation details here will matter a lot, since this type of search is really picky when it comes to performance/recall balance. OpenDistro has already been working on their own version: https://opendistro.github.io/for-elasticsearch/features/knn.... ...so will they switch their API? Perhaps - but the results are going to be very different.
AFAIK when it comes to nearest-neighbor searches Elasticsearch is catching up with OpenDistro k-NN, and not the other way around.
Perhaps, but the point about divergence stands! Elastic won’t be copying the OpenDistro KNN API.
It is pretty clear to me that Elastic is planning to build their ANN features differently than OpenDistro's k-NN implementation, or other plugins modules that extend Easticsearch in similar ways. They now will build on the Apache Lucene capabilities that were collaboratively built "upstream" by a number of individuals, some that work for Amazon and some that work for Elastic.
From the linked issue, it seemed that they were originally planning to develop this as a proprietary feature of Elasticsearch, without contributing the functionality to Apache Lucene, but then changed direction when the Apache Lucene developers (some of which are currently employed to do such work by Amazon) started to build its approximate nearest neighbor (ANN) vector search capabilities. [1]
It's great to see folks that work for Elastic collaborating and building on what is in Apache Lucene to extend the utility of ANN with Hierarchical Navigable Small World Graphs (HNSW) [2]! From this, I think it should be possible to implement an Open Source version of the functionality with a compatible API, if that is something that OpenSearch users seek.
[1] https://issues.apache.org/jira/browse/LUCENE-9004
[2] https://github.com/apache/lucene/pull/250
"it should be possible to implement an Open Source version of the functionality".
Elasticsearch is still open source...just not in a license that Amazon can take advantage of.
Why would most companies need to migrate if they're happy with the features in a past version? Why not stick with it (assuming there are no major security flaws?)
Well, how far is Amazon going to take OpenSearch? Elastic is all in on innovating and building their flagship product forever with significant dedication. AWS just needs something good enough as an offering to go with the rest of their OK cloud services.
I think most people can see both sides of this conflict. But, as you yourself pointed out at the end of your comment, finding a side to pick quickly is important.