I'd say this is more like pytorch is horribly packaged and a nightmare as an eng user to integrate into codebases. All of their assumptions are you use conda and are a datascientist one time doing something in an ipython notebook. Very little consideration is made to how you deploy at scale reliably. It's not really on poetry.
While this is on the one hand at least partially true, it is also the case that as long a poetry can't deal with these (and similar) cases then it cannot really be said to be a candidate for the 'default' dependency management tool. Having people say 'just use poetry' as the go to answer is very bad default advice as long as poetry will fail in a number of important cases.
Based on this comment 5 days ago[0], it's working? I'm not sure didn't dig in too far but based on that comment it seems fair to say that it's not fully Poetry's fault because torch removed hashes (which poetry needs to be effective) for a while only recently adding it back in.
Not sure where I would stand if I fully investigated it tho.
The fact that it only kind of sort of incidentally works some times shows clearly that people who consider PyTorch a vital library for their work aren't the target audience for poetry. Another use case they don't really care about are people for whom compiling C++ etc. is important part of their build process.
Poetry is great if you're developing the sort of software that the poetry devs priorities, but since poetry is only targeting a subset of the python community it will never be a good default tool for all python development, and thus the fragmentation continues.
Every comment in that thread, including the comment you referenced, is wrong.
Due to Poetry's architecture, it can't satisfy all three at the same time:
- the platonic ideal of build isolation and lockfiles
- installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects
- dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere
This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles.
Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. They have something that will not correctly install anything that is dependent itself on PyTorch, which is useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry.
2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.
Sorry if my questions are maybe not well informed (maybe I should delve more into Poetry, I don't really know it), but I don't really understand those points, but I would like to understand them.
What does it mean, build isolation? And how does PyTorch violate that? Or why can't we install PyTorch with build isolation?
What is the issue with lockfiles?
I can understand there might be issues if you need to install a custom accelerator-specific version of PyTorch (although also I don't fully understand what the issues are about this). However, what's with just the standard version, which is suitable for most people, as it comes with CUDA support?
What do you mean by non-interactively? pip install is non-interactively, or not?
What issues are there with the dependencies for PyTorch? Why are they not compatible with the dependencies expressed elsewhere?
How are dependencies defined by requirements.txt different than dependencies defined by pyproject.toml or by setup.py?
If your answer is RTFM, maybe you can point me to some resources. I read a bit through Poetry documentation, but I still don't really understand those issues.
The issue isn't poetry per se, but what simplifying assumptions (that there is a single solution and you can compute it on any system, and that "system effects" can be ignored/handled by wheels, see https://pypackaging-native.github.io/ which covers some of this) it tries to make to solve the solution of what to install.
"Build isolation" is all about limiting what is available at build time (this generally conflicts with inspecting the environment/system), and is vaguely (but no where near sufficient) for reproducible builds (see https://reproducible-builds.org/, which came from linux distros). Lockfiles are similarly an issue because they try to assume they can cover all cases (which they cannot, see e.g. various discussions about sdists for the lockfile PEP). Both these conflict with trying to work out what works for the current system (which is what pytorch tries to do because there are so many different options), or with trying to control what is discovered when trying to do the build on one system but run on another.
But, it does seem to go off and download many GBs of pytorch packages for all possible python, architecture and OS versions which takes ages. Feels very broken at least.
Comments
Despite these comments, Poetry is pretty limited. You cannot install PyTorch with it for example, and the developers do not care: https://github.com/python-poetry/poetry/issues/6409
So basically the single most exciting thing to ever happen to the ecosystem isn’t supported by Poetry.
I'd say this is more like pytorch is horribly packaged and a nightmare as an eng user to integrate into codebases. All of their assumptions are you use conda and are a datascientist one time doing something in an ipython notebook. Very little consideration is made to how you deploy at scale reliably. It's not really on poetry.
It's not really on poetry.
While this is on the one hand at least partially true, it is also the case that as long a poetry can't deal with these (and similar) cases then it cannot really be said to be a candidate for the 'default' dependency management tool. Having people say 'just use poetry' as the go to answer is very bad default advice as long as poetry will fail in a number of important cases.
But you can use poetry for it xd
Based on this comment 5 days ago[0], it's working? I'm not sure didn't dig in too far but based on that comment it seems fair to say that it's not fully Poetry's fault because torch removed hashes (which poetry needs to be effective) for a while only recently adding it back in.
Not sure where I would stand if I fully investigated it tho.
[0] https://github.com/python-poetry/poetry/issues/6409#issuecom...
The fact that it only kind of sort of incidentally works some times shows clearly that people who consider PyTorch a vital library for their work aren't the target audience for poetry. Another use case they don't really care about are people for whom compiling C++ etc. is important part of their build process.
Poetry is great if you're developing the sort of software that the poetry devs priorities, but since poetry is only targeting a subset of the python community it will never be a good default tool for all python development, and thus the fragmentation continues.
Every comment in that thread, including the comment you referenced, is wrong.
Due to Poetry's architecture, it can't satisfy all three at the same time:
- the platonic ideal of build isolation and lockfiles
- installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects
- dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere
This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles.
Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. They have something that will not correctly install anything that is dependent itself on PyTorch, which is useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry.
Here's the facts of the matter:
- 1.2M requirements.txts https://github.com/search?q=path%3A%2F%5Erequirements.txt%24...
- 664k setup.pys https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type...
- setup.pys that reference requirements.txts https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requ... 67.1k
- only 30k pyproject.toml specified with dependencies https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2...
2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.
Sorry if my questions are maybe not well informed (maybe I should delve more into Poetry, I don't really know it), but I don't really understand those points, but I would like to understand them.
What does it mean, build isolation? And how does PyTorch violate that? Or why can't we install PyTorch with build isolation?
What is the issue with lockfiles?
I can understand there might be issues if you need to install a custom accelerator-specific version of PyTorch (although also I don't fully understand what the issues are about this). However, what's with just the standard version, which is suitable for most people, as it comes with CUDA support?
What do you mean by non-interactively? pip install is non-interactively, or not?
What issues are there with the dependencies for PyTorch? Why are they not compatible with the dependencies expressed elsewhere?
How are dependencies defined by requirements.txt different than dependencies defined by pyproject.toml or by setup.py?
If your answer is RTFM, maybe you can point me to some resources. I read a bit through Poetry documentation, but I still don't really understand those issues.
The issue isn't poetry per se, but what simplifying assumptions (that there is a single solution and you can compute it on any system, and that "system effects" can be ignored/handled by wheels, see https://pypackaging-native.github.io/ which covers some of this) it tries to make to solve the solution of what to install.
"Build isolation" is all about limiting what is available at build time (this generally conflicts with inspecting the environment/system), and is vaguely (but no where near sufficient) for reproducible builds (see https://reproducible-builds.org/, which came from linux distros). Lockfiles are similarly an issue because they try to assume they can cover all cases (which they cannot, see e.g. various discussions about sdists for the lockfile PEP). Both these conflict with trying to work out what works for the current system (which is what pytorch tries to do because there are so many different options), or with trying to control what is discovered when trying to do the build on one system but run on another.
Is there a path to declarative package manifests that reach Poetry’s platonic ideals?
Frankly setup.py is a big hack and we should strive to do better.
Got it, thanks for the detailed update. Clearly much more complex.
You can install pytorch with it, I have been doing so for a while.
But, it does seem to go off and download many GBs of pytorch packages for all possible python, architecture and OS versions which takes ages. Feels very broken at least.Agreed, I haven't been too impressed with poetry.