One could argue that small exposure to these metals is what caused us our body to develop defenses against them. If you remove the "small" exposure, you remove the cure, so to speak. Similar to how small quantities of cyanide create an adaptation in your body, but a slightly bigger one would kill you. Minimizing exposure can be net negative for us, as humans, and four our future children.