Not sure this is the right solution for 2 reasons:
First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure.
Second, not all files under version control necessarily belong in a tarball, and not all files in a tarball necessarily need to be under version control.
However, the point that there should be an easy way to track changes in your dependencies is well taken. One possible approach would be that publishing a packge boils down to importing the files that normally would go in your tarball into a community-controlled VCS.
Archlinux has this separation of official packages and the user submitted AUR, which has never been something that is officially supported by design.
Having big names maintain their own VCS is a good thing. There can be a centralized system to index / search packages just for findability. And this centralized system doesn't have to be anything special just like how anyone could use any search engine
First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure.
Why can't you store a (possibly shallow) clone of the repository in the "package" registry?
Second, not all files under version control necessarily belong in a tarball
How likely is it that files under version control but "not belonging in a tarball" (not needed to compile the current version) take so much space to justify making a tarball?
At least if you make a shallow clone.
not all files in a tarball necessarily need to be under version control
Which wouldn't (besides huge files that can be handled with git-lfs and similar)?
If you have configure in mind, let's not do it please.
The risks introduced by including them vastly overcome their convenience, in my opinion.
And maybe let's move towards getting rid of autotools
Why can't you store a (possibly shallow) clone of the repository in the "package" registry?
Yes, that would be an option.
As to your question, you generally want to version control input files, but distribute the generated artifacts. Some possible scenarios would be documentation, data tables in source or binary form, generated code including maybe even configure scripts if you cannot avoid it (though if I had to 'vendor' 3rdparty code that uses autotools, I think I probably would just unpack a tarball and commit the relevant artifacts).
If you distribute generated artifacts you're not really distributing the source (and thus xz can happen)
Even for documentation, you'd think it would be harmless to generate it, but it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files
it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files
Generated HTML files are potentially easier to audit than the scripts/toolchains used to generate them on an end user's machine if you do not pre-generate them.
Off the top of my head, other things I've done is committing RELAX NG *.rnc files, but shipping *.rng files, or generating C header files for various types of data (think `xxd -i` in case of binary files, but also just large chunks of plain text that gets wrapped into a C string).
Sure - but running build scripts on an end-user's machine requires the user to have all relevant tools installed, and isn't exactly reducing the attack surface...
I still like the idea of shipping tarballs that include generated files instead of pulling input files from source control. As mentioned, the first thing that came to mind to make things easier to audit is to stick their contents into a community-controlled VCS.
Sure - but running build scripts on an end-user's machine requires the user to have all relevant tools installed, and isn't exactly reducing the attack surface...
Well, that's what building from source means; you could maybe distribute some compiled files for those who prefer them, and are willing to take the risk.
But you seem to be arguing that having users compile their software themselves doesn't increase their security?
As mentioned, the first thing that came to mind to make things easier to audit is to stick their contents into a community-controlled VCS.
But you seem to be arguing that having users compile their software themselves doesn't increase their security?
On the contrary! I gave examples where source files get generated. This happens in one of two places: Either when the maintainer publishes a new version, or every time an end user builds the package.
I'm arguing for the former, you're arguing for the latter. There are pros and cons to either approach. Some cons for the latter:
First, the build process becomes more brittle, as every end user now has to install the necessary tools. In case of the project containing RELAX NG schemas I alluded to, this would require a recent version of Perl, a Java runtime, and the Trang utility written in Java. The alternative? Just shipping a single XML file.
Second, the build process often doesn't become more, but less auditable: Instead of just reviewing the source files that actually get compiled, you now have to track down how they get generated, and review all the scripts that do so.
Comments
Not sure this is the right solution for 2 reasons:
First, I'm uncomfortable with making a package creator's VCS provider part of the language's module infrastructure.
Second, not all files under version control necessarily belong in a tarball, and not all files in a tarball necessarily need to be under version control.
However, the point that there should be an easy way to track changes in your dependencies is well taken. One possible approach would be that publishing a packge boils down to importing the files that normally would go in your tarball into a community-controlled VCS.
Archlinux has this separation of official packages and the user submitted AUR, which has never been something that is officially supported by design.
Having big names maintain their own VCS is a good thing. There can be a centralized system to index / search packages just for findability. And this centralized system doesn't have to be anything special just like how anyone could use any search engine
Why can't you store a (possibly shallow) clone of the repository in the "package" registry?
How likely is it that files under version control but "not belonging in a tarball" (not needed to compile the current version) take so much space to justify making a tarball? At least if you make a shallow clone.
Which wouldn't (besides huge files that can be handled with git-lfs and similar)?
If you have configure in mind, let's not do it please.
The risks introduced by including them vastly overcome their convenience, in my opinion.
And maybe let's move towards getting rid of autotools
Why can't you store a (possibly shallow) clone of the repository in the "package" registry?
Yes, that would be an option.
As to your question, you generally want to version control input files, but distribute the generated artifacts. Some possible scenarios would be documentation, data tables in source or binary form, generated code including maybe even configure scripts if you cannot avoid it (though if I had to 'vendor' 3rdparty code that uses autotools, I think I probably would just unpack a tarball and commit the relevant artifacts).
If you distribute generated artifacts you're not really distributing the source (and thus xz can happen)
Even for documentation, you'd think it would be harmless to generate it, but it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files
it's far from a given that the tools (e.g. browsers) used to read it are safe from malicious documentation files
Generated HTML files are potentially easier to audit than the scripts/toolchains used to generate them on an end user's machine if you do not pre-generate them.
Off the top of my head, other things I've done is committing RELAX NG *.rnc files, but shipping *.rng files, or generating C header files for various types of data (think `xxd -i` in case of binary files, but also just large chunks of plain text that gets wrapped into a C string).
Hmm, potentially, the problem is that hardly anyone ever audits packages, while at least someone occasionally gives a look at a repository history
Can't the things you list be part of the build scripts?
Sure - but running build scripts on an end-user's machine requires the user to have all relevant tools installed, and isn't exactly reducing the attack surface...
I still like the idea of shipping tarballs that include generated files instead of pulling input files from source control. As mentioned, the first thing that came to mind to make things easier to audit is to stick their contents into a community-controlled VCS.
Well, that's what building from source means; you could maybe distribute some compiled files for those who prefer them, and are willing to take the risk.
But you seem to be arguing that having users compile their software themselves doesn't increase their security?
True
But you seem to be arguing that having users compile their software themselves doesn't increase their security?
On the contrary! I gave examples where source files get generated. This happens in one of two places: Either when the maintainer publishes a new version, or every time an end user builds the package.
I'm arguing for the former, you're arguing for the latter. There are pros and cons to either approach. Some cons for the latter:
First, the build process becomes more brittle, as every end user now has to install the necessary tools. In case of the project containing RELAX NG schemas I alluded to, this would require a recent version of Perl, a Java runtime, and the Trang utility written in Java. The alternative? Just shipping a single XML file.
Second, the build process often doesn't become more, but less auditable: Instead of just reviewing the source files that actually get compiled, you now have to track down how they get generated, and review all the scripts that do so.