I thought when your program/library has a more permissive license you can use a GPL library without changing your license.

No, if it meets the definition of a single work (which is the only time the GPL has an effect on licensing the other material at all), the whole thing must licensed under the GPL. Within certain bounds, to allow compatibility, additional restrictions from the upstream licenses may be imposed, and for things you own you may give additional pernissions (but you cannot add permissions to upstream software included, and downstream licensors under the GPL can remove your additional permissions.)