It's not just a matter of trust. If you verified the JS, you could trust it. The issue is that it's very easy to inject things into an HTML page on the fly. It would be like trying to say that you could trust GnuPG, but you download it from the internet and install it every single time that you use it. That's a huge attack surface.