That's true, and a good point. Perhaps instead of varying the mining difficulty, the reward for confirming a new block could be varied. That ensures that as long as there are enough people interested in mining, you won't have this problem. The miners would still get the transaction fees even if the reward level was at zero. If that kills mining interest enough to have insufficient miners, Bitcoin will someday face a similar problem.