Then we give up our control to them, and in return we are given lives of leisure, luxury, with minimal suffering, the end of disease, etc. Is this such a horrible scenario?

Until the AI overlords need to reclaim space to built another paperclip factory.

Pets also live lives of leisure, in theory. Do all of them do? Are they never abandoned, beaten, tortured for fun, overfed, left alone in hot cars?