Not only did calculators not make the average person great at higher level math when they no longer had to do manual arithmetic

It's even worse than that: calculators can actually make higher level math more difficult (at least for me). I never developed strong manual arithmetic skills because I was a huge pro-calculator partisan in elementary school. When I got to college I really struggled with calculus, because manipulating equations requires arithmetic and that meant I had extra mental workload to operate the calculator.