Exactly. If you are letting an attacker define not just the values of environment variables, but also the names, then you've already lost.

Forget bash. LD_PRELOAD by design lets you load arbitrary code into any child process. Many other environment variables affect libc and other common libraries in arbitrary ways. setuid programs must clear their environment at startup to be safe. Programs like sudo can restore that environment once the user is authorized for root access.