https://clarifai.com/developer/guide/search#search does something very similar as a service; it allows you to ingest numerous images, then feed them through constantly-evolving models and have any number of model-based indices over those images that can answer similarity queries based on previously-unseen inputs. Great to see that there's open-source competition, and that they're focusing on developer productivity (via the tight coupling with Torch) rather than prematurely adding layers of abstraction.
Isn't using pretrained CNN models make embedding biased towards extracting features that were relevant from the dataset (i.e. ImageNet) it learned from? Fine-grained classification uses triplet and siamese networks to learn an embedding based on semantic similarity, but you have to define what is semantically similar in the dataset. I am curious how well pretrain networks generalize for indexing for image search. I think finding papers how pinterest and ebay apply visual search at scale may shed some light on this.
The successive convolutional layers in popular CNN architectures learn representations from very general (edges) to very specific (dog breeds) from the input to the last conv. layer respectively. Depending on how different your new domain is will dictate on what layer you will take the CNN representation from (early layers or later layers) and whether a generic ImageNet model will work at all.
Of course if your new domain is very different than the distribution of the original training data, it is a good technique to fine-tune the network a bit with your data.
Just an extra note, that is the key point why EuclidesDB support multiple models, so you can have for instance a ResNet trained on ImageNet for some images and another ResNet (same architecture) fine-tuned on your data (domain adapted) for another different semantic space. A concrete example is to think in the example of a fashion company who has fine-tuned different models for different product categories:
Model A = fine-tuned to classify between different types of shoes;
Model B = fine-tuned to classify between different t-shirt types;
EuclidesDB can have these two models and you can add/query items into each one of these different models (hence the concept of "model/module space" that is used by EuclidesDB).
yes, this is essentially the Clarifai product: use a CNN (without the softmax) as a feature extractor for images and then do ANN search on them in the embedding space.
Comments
https://clarifai.com/developer/guide/search#search does something very similar as a service; it allows you to ingest numerous images, then feed them through constantly-evolving models and have any number of model-based indices over those images that can answer similarity queries based on previously-unseen inputs. Great to see that there's open-source competition, and that they're focusing on developer productivity (via the tight coupling with Torch) rather than prematurely adding layers of abstraction.
Isn't using pretrained CNN models make embedding biased towards extracting features that were relevant from the dataset (i.e. ImageNet) it learned from? Fine-grained classification uses triplet and siamese networks to learn an embedding based on semantic similarity, but you have to define what is semantically similar in the dataset. I am curious how well pretrain networks generalize for indexing for image search. I think finding papers how pinterest and ebay apply visual search at scale may shed some light on this.
The successive convolutional layers in popular CNN architectures learn representations from very general (edges) to very specific (dog breeds) from the input to the last conv. layer respectively. Depending on how different your new domain is will dictate on what layer you will take the CNN representation from (early layers or later layers) and whether a generic ImageNet model will work at all.
Of course if your new domain is very different than the distribution of the original training data, it is a good technique to fine-tune the network a bit with your data.
Just an extra note, that is the key point why EuclidesDB support multiple models, so you can have for instance a ResNet trained on ImageNet for some images and another ResNet (same architecture) fine-tuned on your data (domain adapted) for another different semantic space. A concrete example is to think in the example of a fashion company who has fine-tuned different models for different product categories:
Model A = fine-tuned to classify between different types of shoes;
Model B = fine-tuned to classify between different t-shirt types;
EuclidesDB can have these two models and you can add/query items into each one of these different models (hence the concept of "model/module space" that is used by EuclidesDB).
yes, this is essentially the Clarifai product: use a CNN (without the softmax) as a feature extractor for images and then do ANN search on them in the embedding space.