The related factors of weight, speed, and fuel consumption would be the obvious downsides to adding armor. Especially considering that a heavier, slower plane is an easier target, thicker skinned or not.

There's a fine balancing point between just enough and too much armor.