design definitely matters. There is the emotional part of design - does it evoke the right emotions just by the way it looks. Then there is the usability part of design. Usability is all about speed

There are other areas, but these are the key ones:

1) learnability -how fast can a first time (new) user figure out how to accomplish the primary tasks 2) efficiency - how fast can a repeat user accomplish the primary tasks 3) memorability - how fast can a user regain proficiency after a period of time

Core to this is understanding the primary tasks. Design is absolutely critical when you think about the essence of it as speed