Of course, more recipients equals more targets for an attacker.

This is especially significant if the recipients have different sized public keys. If you send a message to a 2048-bit key and a 1024-bit key, the message only has 1024 bits of security. So you really do get a weakest link effect.