but it's still considered a middle-class income in the more expensive parts of the country.

Middle class is explicitly not about passive income. If you are living off "safe withdrawal rate", you are wealthy, almost tautologically.

In extreme versions (e.g. some of the FIRE folks) you may be living some version of a "middle class lifestyle", but that's not the same thing. It fundamentally changes the options and opportunities you have.