Don't confuse the conference center for the conferences inside. The conferences can make money (though they often don't) while the center loses money. Either because they can't book enough, or because there is enough competition that you have to lower prices.

Remember that there are only so many businesses (including non-profits) in town that want to rent the place. When people are coming from out of town what difference does the town make? Thus it is hard to fill your conference center.