Google Phone with Location Disabled

This is just a random thing that annoys me to no end. I have my Google phone configured to disable location almost all of the time. Google wants me to enable my location all of the time. Recently I enabled my location for a single app, and then when I was searching on Google maps later I noticed it was still enabled. Maybe this is no surprise to anyone, and I admit I was only partially surprised. But come on if my location is disabled why leave it enabled? Also, you can't disable the location from the pull down screen when the phone is locked... I realize this is for "security" so the bad guys can't disable your location, but in reality it's just for Google too have more data about you.