It's actually important in two major ways. First, conversion rates have been known to increase in correlation with site speed [1]. Second, mobile devices are often on terrible connections.

1: http://blog.mozilla.org/metrics/2010/03/31/firefox-page-load...