I spent a week or so pouring through valgrind and profile output optimising the very heart of the code for a Monte Carlo simulation at a major investment bank. Some of the lines of code I wrote in that week will have been executed billions of times per day since I pushed them back in 2007 or so. They are still being used.

I got a good overall speedup (over 5% in code that was already pretty well-optimised) but by far the most valuable optimisation I did was one of the most basic - hoisting variables out of for loops. This kind of thing

    for(int i=0; i<something; ++i) {
       int x=0;
       //...do something with x  
    }

changes to...

    int x;
    for(int i=0; i<something; ++i) {
       x=0;
       //...do something with x  
    }

There are slightly more subtle versions of that, but at the time none of the compilers we were using (gcc 3.2, solaris 8 C++ compiler or Visual C++ 7) could optimise that without you doing the hoist yourself.