why didn't chips suddenly jump from 100um to 100nm scale?

Manufacturing processes. As you go to smaller scales, the investment required to do so increases, and the error rate as well. When you produce millions of units you want to have a low reject rate, and your process needs to be fully controlled. That's why there's always a gap between what's technically possible and what makes sense in a industrial context.