You can't know what to optimize without taking into account how the application is going to be used. And even knowing that, profiling and cost/benefit analyses are the ultimate guide. I think the best rule of thumb is to not worry too much about optimizing everything on the first pass, but be careful not to cut off future optimizations and prematurely lock in poor performance. I always like having my options open.