Python's fine if the heavy lifting is in a C extension. I use Cython for this; others prefer just numpy, maybe something like numba. But you can't just have a Python list of floats.

Java, C++, Scala and Julia are all popular choices. Go is probably fine too, although I know less about it.