Several reasons - computational effort, effort it takes to reproduce results on complex datasets, academic publishing model.
Academic research is often times about making incremental steps and limiting uncertainty.
Making something work for MNIST is already so much work, researchers don’t have the time, money, and energy to run experiments for 10 datasets.
Complex datasets are much harder to get a proper model trained on due to increased complexity - larger images, tasks, classes, etc.
Also, as soon as you run your experiments on more datasets, you create an opportunity for reviewers to take you down - “why didn’t you test it on this other dataset?”
Comments
Several reasons - computational effort, effort it takes to reproduce results on complex datasets, academic publishing model.
Academic research is often times about making incremental steps and limiting uncertainty.
Making something work for MNIST is already so much work, researchers don’t have the time, money, and energy to run experiments for 10 datasets.
Complex datasets are much harder to get a proper model trained on due to increased complexity - larger images, tasks, classes, etc.
Also, as soon as you run your experiments on more datasets, you create an opportunity for reviewers to take you down - “why didn’t you test it on this other dataset?”