Every comment in that thread, including the comment you referenced, is wrong.
Due to Poetry's architecture, it can't satisfy all three at the same time:
- the platonic ideal of build isolation and lockfiles
- installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects
- dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere
This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles.
Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. They have something that will not correctly install anything that is dependent itself on PyTorch, which is useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry.
2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.
Sorry if my questions are maybe not well informed (maybe I should delve more into Poetry, I don't really know it), but I don't really understand those points, but I would like to understand them.
What does it mean, build isolation? And how does PyTorch violate that? Or why can't we install PyTorch with build isolation?
What is the issue with lockfiles?
I can understand there might be issues if you need to install a custom accelerator-specific version of PyTorch (although also I don't fully understand what the issues are about this). However, what's with just the standard version, which is suitable for most people, as it comes with CUDA support?
What do you mean by non-interactively? pip install is non-interactively, or not?
What issues are there with the dependencies for PyTorch? Why are they not compatible with the dependencies expressed elsewhere?
How are dependencies defined by requirements.txt different than dependencies defined by pyproject.toml or by setup.py?
If your answer is RTFM, maybe you can point me to some resources. I read a bit through Poetry documentation, but I still don't really understand those issues.
The issue isn't poetry per se, but what simplifying assumptions (that there is a single solution and you can compute it on any system, and that "system effects" can be ignored/handled by wheels, see https://pypackaging-native.github.io/ which covers some of this) it tries to make to solve the solution of what to install.
"Build isolation" is all about limiting what is available at build time (this generally conflicts with inspecting the environment/system), and is vaguely (but no where near sufficient) for reproducible builds (see https://reproducible-builds.org/, which came from linux distros). Lockfiles are similarly an issue because they try to assume they can cover all cases (which they cannot, see e.g. various discussions about sdists for the lockfile PEP). Both these conflict with trying to work out what works for the current system (which is what pytorch tries to do because there are so many different options), or with trying to control what is discovered when trying to do the build on one system but run on another.
Comments
Every comment in that thread, including the comment you referenced, is wrong.
Due to Poetry's architecture, it can't satisfy all three at the same time:
- the platonic ideal of build isolation and lockfiles
- installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects
- dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere
This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles.
Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. They have something that will not correctly install anything that is dependent itself on PyTorch, which is useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry.
Here's the facts of the matter:
- 1.2M requirements.txts https://github.com/search?q=path%3A%2F%5Erequirements.txt%24...
- 664k setup.pys https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type...
- setup.pys that reference requirements.txts https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requ... 67.1k
- only 30k pyproject.toml specified with dependencies https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2...
2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.
Sorry if my questions are maybe not well informed (maybe I should delve more into Poetry, I don't really know it), but I don't really understand those points, but I would like to understand them.
What does it mean, build isolation? And how does PyTorch violate that? Or why can't we install PyTorch with build isolation?
What is the issue with lockfiles?
I can understand there might be issues if you need to install a custom accelerator-specific version of PyTorch (although also I don't fully understand what the issues are about this). However, what's with just the standard version, which is suitable for most people, as it comes with CUDA support?
What do you mean by non-interactively? pip install is non-interactively, or not?
What issues are there with the dependencies for PyTorch? Why are they not compatible with the dependencies expressed elsewhere?
How are dependencies defined by requirements.txt different than dependencies defined by pyproject.toml or by setup.py?
If your answer is RTFM, maybe you can point me to some resources. I read a bit through Poetry documentation, but I still don't really understand those issues.
The issue isn't poetry per se, but what simplifying assumptions (that there is a single solution and you can compute it on any system, and that "system effects" can be ignored/handled by wheels, see https://pypackaging-native.github.io/ which covers some of this) it tries to make to solve the solution of what to install.
"Build isolation" is all about limiting what is available at build time (this generally conflicts with inspecting the environment/system), and is vaguely (but no where near sufficient) for reproducible builds (see https://reproducible-builds.org/, which came from linux distros). Lockfiles are similarly an issue because they try to assume they can cover all cases (which they cannot, see e.g. various discussions about sdists for the lockfile PEP). Both these conflict with trying to work out what works for the current system (which is what pytorch tries to do because there are so many different options), or with trying to control what is discovered when trying to do the build on one system but run on another.
Is there a path to declarative package manifests that reach Poetry’s platonic ideals?
Frankly setup.py is a big hack and we should strive to do better.
Got it, thanks for the detailed update. Clearly much more complex.