leisure time, which was really something only rich people had, once upon a time)

Disagree, its been shown that hunter gatherer tribes in various locations generally only need to work for about 4h a day on average.

I believe the “there is only time for work” stuff turned up as power structures developed. A pair of busy hands has less time to plot to overthrow the overlords, after all.