Yeah maybe I was thinking of scipy. I've only built numpy and scipy from source once, and it was to install scikit-learn, which has both of them as a dependency. I had to install gfortran first.