Maybe someone from Dropbox can add more color to this and explain what other options they considered.
As it stands, I still can't find what I need in Dropbox. And never could. From reading this article I'd think searching for a basic keyword like "dog" or "ship" or "runner" would yield some results from my tens of thousands of photos, yet I get nothing (nothing relevant, at least).
Edit: On second reading, this is only available to Dropbox Pro and Business users. I hope they roll this out to other paying users soon.
Search by image is conceptually easier because you don't have to map between text and images, but it's a very different product. It is something we've considered.
Encoding words and images into the same space and doing ANN is kind of what the current system is, if you look at it right. The ANN is framed in terms of similarity rather than distance -- and is approximate because of the sparseness approximation. But the big difference from the papers you linked is what we use as the encodings: not the traditional penultimate layer of a network, but classifier scores for images and projected word vectors for text. This gives us a space with semantically meaningful dimensions, which lets us build the system without a large multimodal training set; our text and image models are independently trained on different datasets.
Comments
That would work for sure if they wanted to add a "more images like this" feature. Here's an example using the SqueezeNet model and Pinecone (where scale isn't an issue): https://www.pinecone.io/docs/examples/image-similarity-searc...
Encoding both word and image embeddings into the same index then doing ANN on that index might also work. See this example of text-to-image retrieval: https://paperswithcode.com/task/texture-image-retrieval
Maybe someone from Dropbox can add more color to this and explain what other options they considered.
As it stands, I still can't find what I need in Dropbox. And never could. From reading this article I'd think searching for a basic keyword like "dog" or "ship" or "runner" would yield some results from my tens of thousands of photos, yet I get nothing (nothing relevant, at least).
Edit: On second reading, this is only available to Dropbox Pro and Business users. I hope they roll this out to other paying users soon.
Search by image is conceptually easier because you don't have to map between text and images, but it's a very different product. It is something we've considered.
Encoding words and images into the same space and doing ANN is kind of what the current system is, if you look at it right. The ANN is framed in terms of similarity rather than distance -- and is approximate because of the sparseness approximation. But the big difference from the papers you linked is what we use as the encodings: not the traditional penultimate layer of a network, but classifier scores for images and projected word vectors for text. This gives us a space with semantically meaningful dimensions, which lets us build the system without a large multimodal training set; our text and image models are independently trained on different datasets.
Interesting. Thanks for responding.
1. Did you look at CLIP? it provides a common (to images & text) embedding.
2. Do your models need specialized training (vs. open models)?