Think of a very very elliptical orbit, one that's nearly a line and consider what happens when you take a bit of energy off the orbiting object. At perigee, when the energy is almost entirely kinetic, you're mostly just changing the velocity, rather than altitude of the object. At apogee, when the energy is almost entirely potential, it's the other way round - the change is almost entirely one of altitude. This is further stretched by the much more 'visible' effect of potential energy changes on altitude than of kinetic energy changes on velocity.