The encrypted ballots are all public and it isn't assumed they they're anonymous (eg. it's safe to post a picture of your RHS ballot on Facebook).

If we just directly decrypted them and published the results, they would allow someone to prove who they had voted for.

So instead, we shuffle them to anonymise them, then decrypt them to avoid being able to link an encrypted input vote to an output vote.