I've been in an anechoic chamber and it's not so much the lack of sound but the intense visual shock. You're usually in a room with incredible depth levels, sharp angles and so on. It can induce a feeling of sea-sickness.

The absolute silence exacerbates the issue but it's not the only factor at play.