That's not just trust in them, it's partly trust in the community that if they shipped a malicious browser, say, someone would notice and cry foul.

Offering source that doesn't match the binaries doesn't stop someone monitoring their own network traffic. And if you recompile the binaries with the same settings, the hashes should match.