You are assuming that with or without bike infrastructure, there is a fixed number of cyclists.

Many drivers either believe there are no cyclists; or that the number of cyclists should be reduced to zero, and they should drive instead, like normal people.

Also, everything is zero-sum, so any money spent on bike infrastructure is less money spent to widen the freeways or subsidize parking.