It was Python trying to figure out where its modules and such were. It would try a bunch of likely-seeming directories and get ENOENT until it finally hit the right place. Most HPC systems are kind of expecting that you'll ship a single compiled C/C++ binary to each node, each of which may read a file or two of input from the fileserver. With Python, we had a thousand times as many requests and it just killed the poor thing.
This was on an IBM BlueGene. You can make your own call on the design quality of the system.
My solution to this problem was to build Python so that it finds all of its dependencies in one directory. This project
https://github.com/wavetossed/pybuild has the build script that I used on Ubuntu Lucid LTS. In addition this build ends up being portable to any Linux distro because all of the dependencies are in one directory tree where you can easily create a tarball.
Comments
Python: How to bring your supercomputer to a crawl as 1,000 nodes all try to access hundreds of non-existent files on the same network share.
Sounds like a poorly designed system rather than a Python problem.
It was Python trying to figure out where its modules and such were. It would try a bunch of likely-seeming directories and get ENOENT until it finally hit the right place. Most HPC systems are kind of expecting that you'll ship a single compiled C/C++ binary to each node, each of which may read a file or two of input from the fileserver. With Python, we had a thousand times as many requests and it just killed the poor thing.
This was on an IBM BlueGene. You can make your own call on the design quality of the system.
My solution to this problem was to build Python so that it finds all of its dependencies in one directory. This project https://github.com/wavetossed/pybuild has the build script that I used on Ubuntu Lucid LTS. In addition this build ends up being portable to any Linux distro because all of the dependencies are in one directory tree where you can easily create a tarball.
the shit would happen if you abuse LD_LIBRARY_PATH or PATH for C/C++ shared libraries.
Many bad python code witll sys.path.insert() which would also cause this.
By bad system I didn't mean your awesome IBM BlueGene, but the running system happened to use Python as language.