Ideally pose and lighting wouldn't matter as much as it currently does.

I think using a better model to produce feature vectors could achieve this, or perhaps even finetuning the feature model to match human preferences.