optimized libraries could be written in any language

Well, at least in anything that has really performant compilers. That's currently true for C, Fortran and Assembly, to some extent also C++. Between the first three, Fortran is by far the easiest to use for numerics - its syntax is in fact not much more difficult than Matlab. It's clunky in anything but numerical code (such as when you try to create a nice API or when you try to have generic methods), but at the end it still wins.