All we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is that?

A lot. Parallel optimization is an art form. These models are trained on static datasets, they can't intervene in the environment to infer causal relations, so they need legs and hands.