I just passed on a job for a quality company in the Bay area because they wouldn't budge on remote work

You're assuming they don't trust remote employees? They may just not want remote employees.

It's a perfectly valid stance for a company to say, "You know what, remote work just isn't for us." They don't need to justify it any more than you need to justify your preference for remote work.

It's not right or wrong, it's just a preference.