I did my masters in astrophysics, specialised from being a general physicist for my bachelors.
I was reading the piece, waiting for the punchline of what kind of unholy beast of a workstation she was using, and wasn’t disappointed.
Thing is, physicists, heck, scientists in general, are not programmers. Fortran 77 and python are pretty much the only shows in town - and your usual data crunching script will be huge, procedural, in log time, and will eat mountains of memory while making disks thrash as hard as humanly possible.
For instance, I helped out a postdoc who I shared a lab with with his ephemeris calculator - it’d take in a series of fits images, and it’d tell you the ephemeris of whatever object you chose - by editing the source, and putting in the x/y of the object in the first frame.
Thing was, it took all night to do this for a single object from half a dozen frames. Most of the time was spent opening and closing each file to read each pixel, and then stuffing those pixels into a gigantic array, and writing that array to text files, and then re-reading those files, and then doing matrix multiplication and all sorts of amazingly baroque stuff that must have seemed like a good idea at the time. He was running it on a monster (for the time) of a workstation, with 64GB of ram and several TB of storage.
I banged together an app in C++ for him with a basic tcl/tk gui, and what had taken him a day of setup and a night of processing and an ungodly machine instead took about as long as it took for him to click on an object and click “go” - on my creaking laptop with 128mb of ram and a 1.4gb hard drive.
This was far from singular - after this, I found myself being “the guy” to talk to about your slow scripts - and that was basically every script in the department.
So no, not better, considerably worse, and I can’t help but think that having a more cross-disciplinary approach to science (embed tech people!) would yield benefits across the board.
I feel you on this. Having done a lot of Python consulting for engineers and scientists, it is absolutely the case that most non-programmers have -zero- model of what I/O latency and bandwidth limitations look like. They are looking at programming APIs, and their mental models can include concepts like files and even byte layouts within files. But they generally have no working model of how a physical computer actually implements those things.
I've definitely seen file I/O in the middle of FORTRAN loops. Entirely correct from a functional perspective, and total disaster from an actual runtime perspective.
But they generally have no working model of how a physical computer actually implements those things.
To be fair, how many blog posts have been written over the years on programmers doing ridiculous things with SQL, or network IO, or file IO, etc. etc.? If trained programmers struggle with these things, I'm willing to give non-programmers some slack.
you are describing my job security. I've had a career spanning 25 years speeding up other peoples scripts.
For example once $LARGE_BIOTECH retired its inhouse supercomputer (A TruCluster composed of multiple GS1280s, the epitome of classical UNIX power) and replaced it with Linux and NFS. The principal engineer noticed his pipeline never finished on Linux but how no idea where to start.
Within five minutes I noticed that GNU/Linux sort uses $TMPDIR for large sorts (multipass). They had pointed TMPDIR at an NFS mount and were doing a multipass sort over NFS. On TruCluster, tmp pointed to an ultrafast cluster diskl.
Comments
I did my masters in astrophysics, specialised from being a general physicist for my bachelors.
I was reading the piece, waiting for the punchline of what kind of unholy beast of a workstation she was using, and wasn’t disappointed.
Thing is, physicists, heck, scientists in general, are not programmers. Fortran 77 and python are pretty much the only shows in town - and your usual data crunching script will be huge, procedural, in log time, and will eat mountains of memory while making disks thrash as hard as humanly possible.
For instance, I helped out a postdoc who I shared a lab with with his ephemeris calculator - it’d take in a series of fits images, and it’d tell you the ephemeris of whatever object you chose - by editing the source, and putting in the x/y of the object in the first frame.
Thing was, it took all night to do this for a single object from half a dozen frames. Most of the time was spent opening and closing each file to read each pixel, and then stuffing those pixels into a gigantic array, and writing that array to text files, and then re-reading those files, and then doing matrix multiplication and all sorts of amazingly baroque stuff that must have seemed like a good idea at the time. He was running it on a monster (for the time) of a workstation, with 64GB of ram and several TB of storage.
I banged together an app in C++ for him with a basic tcl/tk gui, and what had taken him a day of setup and a night of processing and an ungodly machine instead took about as long as it took for him to click on an object and click “go” - on my creaking laptop with 128mb of ram and a 1.4gb hard drive.
This was far from singular - after this, I found myself being “the guy” to talk to about your slow scripts - and that was basically every script in the department.
So no, not better, considerably worse, and I can’t help but think that having a more cross-disciplinary approach to science (embed tech people!) would yield benefits across the board.
I feel you on this. Having done a lot of Python consulting for engineers and scientists, it is absolutely the case that most non-programmers have -zero- model of what I/O latency and bandwidth limitations look like. They are looking at programming APIs, and their mental models can include concepts like files and even byte layouts within files. But they generally have no working model of how a physical computer actually implements those things.
I've definitely seen file I/O in the middle of FORTRAN loops. Entirely correct from a functional perspective, and total disaster from an actual runtime perspective.
To be fair, how many blog posts have been written over the years on programmers doing ridiculous things with SQL, or network IO, or file IO, etc. etc.? If trained programmers struggle with these things, I'm willing to give non-programmers some slack.
you are describing my job security. I've had a career spanning 25 years speeding up other peoples scripts.
For example once $LARGE_BIOTECH retired its inhouse supercomputer (A TruCluster composed of multiple GS1280s, the epitome of classical UNIX power) and replaced it with Linux and NFS. The principal engineer noticed his pipeline never finished on Linux but how no idea where to start.
Within five minutes I noticed that GNU/Linux sort uses $TMPDIR for large sorts (multipass). They had pointed TMPDIR at an NFS mount and were doing a multipass sort over NFS. On TruCluster, tmp pointed to an ultrafast cluster diskl.