I dont think its really the cognitive effort that I find annoying but rather the time it actually takes to move my hands from the keyboard to the mouse.

I am firmly in the know thy shortcuts camp and find anything that forces me to move hands away from the keyboard has a negative effect in terms of my productivity. With a mouse and keyboard at least your hands are moving in the same plane.