But how are you handling those system dependencies (is R and its various packages one?), and how do you know that the packages you install/manage via the python ecosystem work with them?

I think the reason Python has so many things is simply because most other languages throw their arms up and say "system stuff, not my problem" (rust is an excellent example of this, the build.rs is basically the same thing as a setup.py, except less standard, and currently lacks from what I've seen is any kind of systematic solution like cibuildwheel), whereas Python has always been trying to do something to address it.