Banishment is most certainly a punishment. Just imagine never being allowed to speak with almost all the people you met in your entire life.

Hunter Gatherers also find a strength and safety in numbers, even if only in avoiding the fatility of personal "catastrophes" like a broken limb.