"If you did that though wouldn't people still feel that it was unfair? (The frame of reference would just be lowered)."

I think it's reasonable to assume that the frame of reference also includes other properties, at least to some extent.