Maybe. If your build is too slow, fix it. But pre-emptively microoptimising your build time is as bad as pre-emptively microoptimising anything else. Set a build time budget and don't worry about it unless and until you see a risk of exceeding that budget.