I'm sure it's a problem, and the degree of "problem" depends on the importance of the mail.

I think the real problem is that we are delegating to google the arbitrary decision of what is spam or not. Given that their core business in great part relies on syphoning info from users' mails, they have zero incentive to let users host their own servers and potentially bypass that data collection, and an actual incemtive to block as many domains as they can, for the sake of "safety"