but if we oversimplify things and assume that every synapse can be modeled as a single parameter in a weight matrix

Which, it probably can't... but offsetting those simplifications and 4-20x difference is the massive difference in how quickly those synapses can be activated.