One reason is that versioning is really hard. Now that Python comes standard with Linux distros, and in fact some of the built in tools are written in Python, it becomes really hard to upgrade. So you can never touch the version that your OS ships with or you'll break some part of the OS. But those are usually years behind and developers generally want the new hotness so you end up installing newer versions elsewhere. There are tools built around this like virtualenv but you end up with a bunch of different versions of Python installed in different places, and then installing modules becomes something that happens outside of your system's package management software so you can't track dependencies etc. And python itself has really weak package management, or rather several competing and incompatible systems but you'll download modules that use each of them so you end up with all of them. At least Ruby and Perl were able to standardise on one each.
So very much this. As a developer, I just love "pip install whatever", but as a sysadmin, it makes me cringe every time - how do I manage this? What happens if I have to reinstall? What do you mean, you want to put files in /usr that aren't known by the packaging system that came with the OS? And other langs seem to be following the same painful path (cf Ruby gems, which I don't believe solve these problems any better than Python). I first encountered this in ROS (http://ros.org), which while really awesome, always felt like a big pile of Python spitballed together.
I completely agree. I've worked at the large companies where we were stuck with Python 2.5 because of LTS versions of distros in production (around 2010).
I work at Continuum using our Python distribution (Anaconda[1] and Miniconda[2]). We sell it as the Data Scientist's answer to python, but as a sysadmin, I really think it is the answer administrators have been looking for. It installs into one directory that can be blown away if needed. It provides virtualenv-like environments except it you can isolate different versions of python or any binary. You can "activate" your environment, just by setting your PATH to find the env's bin first.
The conda package manager installs binary packages, so you don't need to compile c-extensions on each machine. Conda allows you to build all of your packages for each arch. Binstar is our alpha package hosting service that lets you host packages and it has a multi-platform build service (super-alpha). And you aren't limited to Binstar if you want to host your own repos -- the commands are built into conda.
If you want to distribute non-python binaries, you can do that as well. I'm using conda to install node and node packages. I really like using conda for node binaries, because I can build them using npm and then I have a way to push versioned node binaries to production. (Internally, we've built most of the R packages, postgres, mysql, mongo, nginx as conda packages to provide non-root users a way to install normally root-only software).
What are the "several competing and incompatible systems"? I haven't used anything other than pip, ever. I know of easy_install, but as far as I know pip is an outright total replacement for it.
The tools installed by pip won't usually work with the old versions of Python found by default on some systems. They break because they want a new Python feature, and that makes you want to upgrade and that... you can't do that because it will break dependencies at system level.
And then you want to install a new Python side by side and you sort of do that but /usr/bin/python still points to the old version and you have to be explicit with /usr/bin/python2.6 in your scripts. But then some scripts will work and some still don't and you make /usr/bin/python point to /usr/bin/python2.6 and now you really broke yum.
And all you can do now is hate python because it gave you all these problems.
As a sysadmin, I hated java way more than python :-)
At least, with java, no one even attempts to use distro package tools to manage libraries, packages and dependencies. Can you see a sysadmin using a syslog replacement implemented in java, requiring java8, along with a webserver only running on java5?
With python, some tools (those big enough to be packaged) are no problem. Some random tool from pip -- not so much -- now you have to patch that python binary, those python dependencies etc... and not just track security.your-distro.org.
It is certainly possible to deploy your system in a similar way you deploy your apps (take on 100% of the burden of maintaining the whole thing) -- but normally you want the "system" to be a stable foundation on which to erect your rickety duct-taped mono-jvm-python-ruby app-stack with the help of puppet/chef/cfengine/salt/ansible... you don't want pip/hackage/gem/npm to manage your system. You might trust it to manage a single application.
It should be very easy. Things are never installed system-wide -- no cross-app conflicts. When starting an application, either you manually specify each file or directory to include, or you just package every dependency into your executable jar file.
This system administrator hates Java almost as much as Windows. I'm still enraged at the old lie "compile once, run everywhere". Just look at what Java does to the Red Hat "alternatives" system, although I can't imagine any prettier body cast around that catastrophe.
Reading your comment the irony occurs to me that the best solution to "compile once" has been real VMs. Not Java VMs, but full god damned fucking virtual machines running the OS and environment you determine should run in them.
You've still got to provision those, but, well, they're individual and isolated and tend not to go hammering into one another. Other than consuming all available system resources.
Oddly: IBM got this right with VM ... 40 years ago?
I thought there was something going on with Python v2 & v3 that would solve all of this? Then v3 was shelved or something? I saw an older post here on YC about it, but it was above me since I'm mostly a Windows admin :(
Comments
One reason is that versioning is really hard. Now that Python comes standard with Linux distros, and in fact some of the built in tools are written in Python, it becomes really hard to upgrade. So you can never touch the version that your OS ships with or you'll break some part of the OS. But those are usually years behind and developers generally want the new hotness so you end up installing newer versions elsewhere. There are tools built around this like virtualenv but you end up with a bunch of different versions of Python installed in different places, and then installing modules becomes something that happens outside of your system's package management software so you can't track dependencies etc. And python itself has really weak package management, or rather several competing and incompatible systems but you'll download modules that use each of them so you end up with all of them. At least Ruby and Perl were able to standardise on one each.
So very much this. As a developer, I just love "pip install whatever", but as a sysadmin, it makes me cringe every time - how do I manage this? What happens if I have to reinstall? What do you mean, you want to put files in /usr that aren't known by the packaging system that came with the OS? And other langs seem to be following the same painful path (cf Ruby gems, which I don't believe solve these problems any better than Python). I first encountered this in ROS (http://ros.org), which while really awesome, always felt like a big pile of Python spitballed together.
And lo, Python became Perl, and the cycle was complete.
I completely agree. I've worked at the large companies where we were stuck with Python 2.5 because of LTS versions of distros in production (around 2010).
I work at Continuum using our Python distribution (Anaconda[1] and Miniconda[2]). We sell it as the Data Scientist's answer to python, but as a sysadmin, I really think it is the answer administrators have been looking for. It installs into one directory that can be blown away if needed. It provides virtualenv-like environments except it you can isolate different versions of python or any binary. You can "activate" your environment, just by setting your PATH to find the env's bin first.
The conda package manager installs binary packages, so you don't need to compile c-extensions on each machine. Conda allows you to build all of your packages for each arch. Binstar is our alpha package hosting service that lets you host packages and it has a multi-platform build service (super-alpha). And you aren't limited to Binstar if you want to host your own repos -- the commands are built into conda.
If you want to distribute non-python binaries, you can do that as well. I'm using conda to install node and node packages. I really like using conda for node binaries, because I can build them using npm and then I have a way to push versioned node binaries to production. (Internally, we've built most of the R packages, postgres, mysql, mongo, nginx as conda packages to provide non-root users a way to install normally root-only software).
[1] Anaconda: http://continuum.io/downloads [2] Just python: http://conda.pydata.org/miniconda.html#miniconda
What are the "several competing and incompatible systems"? I haven't used anything other than pip, ever. I know of easy_install, but as far as I know pip is an outright total replacement for it.
The tools installed by pip won't usually work with the old versions of Python found by default on some systems. They break because they want a new Python feature, and that makes you want to upgrade and that... you can't do that because it will break dependencies at system level.
And then you want to install a new Python side by side and you sort of do that but /usr/bin/python still points to the old version and you have to be explicit with /usr/bin/python2.6 in your scripts. But then some scripts will work and some still don't and you make /usr/bin/python point to /usr/bin/python2.6 and now you really broke yum.
And all you can do now is hate python because it gave you all these problems.
And I can't imagine what it is like to deploy Java apps if you have no actual familiarity with the JVM ecosystem, either.
As a sysadmin, I hated java way more than python :-)
At least, with java, no one even attempts to use distro package tools to manage libraries, packages and dependencies. Can you see a sysadmin using a syslog replacement implemented in java, requiring java8, along with a webserver only running on java5?
With python, some tools (those big enough to be packaged) are no problem. Some random tool from pip -- not so much -- now you have to patch that python binary, those python dependencies etc... and not just track security.your-distro.org.
It is certainly possible to deploy your system in a similar way you deploy your apps (take on 100% of the burden of maintaining the whole thing) -- but normally you want the "system" to be a stable foundation on which to erect your rickety duct-taped mono-jvm-python-ruby app-stack with the help of puppet/chef/cfengine/salt/ansible... you don't want pip/hackage/gem/npm to manage your system. You might trust it to manage a single application.
It should be very easy. Things are never installed system-wide -- no cross-app conflicts. When starting an application, either you manually specify each file or directory to include, or you just package every dependency into your executable jar file.
What is your point? That administrators really hate Java too?
This system administrator hates Java almost as much as Windows. I'm still enraged at the old lie "compile once, run everywhere". Just look at what Java does to the Red Hat "alternatives" system, although I can't imagine any prettier body cast around that catastrophe.
Reading your comment the irony occurs to me that the best solution to "compile once" has been real VMs. Not Java VMs, but full god damned fucking virtual machines running the OS and environment you determine should run in them.
You've still got to provision those, but, well, they're individual and isolated and tend not to go hammering into one another. Other than consuming all available system resources.
Oddly: IBM got this right with VM ... 40 years ago?
https://en.wikipedia.org/wiki/VM_(operating_system)
That's a challenge with _most_ scripting languages.
It used to be a major PITA with Perl. I've run into it with Ruby (and the proposed solutions are even more horrible: RPM).
And yes, Devs absolutely want teh neu hawtness.
I thought there was something going on with Python v2 & v3 that would solve all of this? Then v3 was shelved or something? I saw an older post here on YC about it, but it was above me since I'm mostly a Windows admin :(