Hard to say without benchmarking. Unless the compilation process can be parallelised (as in "lots of files without interdependencies"), you'll find little use for extra cores beyond the average number of simultaneous possible compilations.

This is all very project-dependent - you might even optimise your build process by moving things around to reduce these dependencies (facilitating parallel compiles).