There's definitely some truth to how (at least some of) e-commerce sites could be easily sped up. Part of the problem is that you have a combination of legacy stack, organizational inertia, lack of know-how and resource outsourcing (so 3rd party vendors owns parts of the stack).

And specifically, wrt caching and varnish, as soon as you put something in your cart and you're cookie'd pages you visit are not cacheable.