Glad to inspire and go ahead and use this idea. By the way my full name is Kaveh Hadjari.
Yes you would still would need a good image classifier like ResNet 34 to actually extract usuable features which could then be used as input to a RNN model.
An important point if you would try to actually implement such a model, don't train the image classifier together with the RNN as that would probably make the training unnecessarily slow with the end result of either not making a dent in the weights of the ResNet or simply overfitting the relatively small and inappropriate training set that is provided by Yelp for this competition. Instead use the ResNet classifier in a preprocessing step to transform the images to the activations of a later layer in the ResNet and use those as the input for the RNN model, note when training the RNN you should persist these activations as you would otherwise would unnecessarily need to run forward prop multiple times through the ResNet for each image. In this manner you get the benefit of the ResNet which is great at extracting features from images and the ability to train the RNN model much more quickly.
Comments
Glad to inspire and go ahead and use this idea. By the way my full name is Kaveh Hadjari.
Yes you would still would need a good image classifier like ResNet 34 to actually extract usuable features which could then be used as input to a RNN model.
An important point if you would try to actually implement such a model, don't train the image classifier together with the RNN as that would probably make the training unnecessarily slow with the end result of either not making a dent in the weights of the ResNet or simply overfitting the relatively small and inappropriate training set that is provided by Yelp for this competition. Instead use the ResNet classifier in a preprocessing step to transform the images to the activations of a later layer in the ResNet and use those as the input for the RNN model, note when training the RNN you should persist these activations as you would otherwise would unnecessarily need to run forward prop multiple times through the ResNet for each image. In this manner you get the benefit of the ResNet which is great at extracting features from images and the ability to train the RNN model much more quickly.