Re: make dist using git archive

Hi,

On Mon, Jan 22, 2024 at 3:32 PM Peter Eisentraut <peter(at)eisentraut(dot)org> wrote:
>
> One of the goals is to make the creation of the distribution tarball
> more directly traceable to the git repository. That is why we removed
> the "make distprep" step.
>
> Here I want to take another step in that direction, by changing "make
> dist" to directly use "git archive", rather than the custom shell script
> it currently runs.
>
> The simple summary is that it would run
>
> git archive --format tar.gz --prefix postgresql-17devel/ HEAD -o
> postgresql-17devel.tar.gz
>
> (with appropriate version numbers, of course), and that's the tarball we
> would ship.
>
> There are analogous commands for other compression formats.
>
> The actual command gets subtly more complicated if you need to run this
> in a separate build directory. In my attached patch, the make version
> doesn't support vpath at the moment, just so that it's easier to
> understand for now. The meson version is a bit hairy.
>
> I have studied and tested this quite a bit, and I have found that the
> archives produced this way are deterministic and reproducible, meaning
> for a given commit the result file should always be bit-for-bit identical.
>
> The exception is that if you use a git version older than 2.38.0, gzip
> records the platform in the archive, so you'd get a different output on
> Windows vs. macOS vs. "UNIX" (everything else). In git 2.38.0, this was
> changed so that everything is recorded as "UNIX" now. This is just
> something to keep in mind. This issue is specific to the gzip format,
> it does not affect other compression formats.
>
> Meson has its own distribution building command (meson dist), but opted
> against using that. The main problem is that the way they have
> implemented it, it is not deterministic in the above sense. (Another
> point is of course that we probably want a "make" version for the time
> being.)
>
> But the target name "dist" in meson is reserved for that reason, so I
> needed to call the custom target "pgdist".
>
> I did take one idea from meson: It runs a check before git archive that
> the checkout is clean. That way, you avoid mistakes because of
> uncommitted changes. This works well in my "make" implementation. In
> the meson implementation, I had to find a workaround, because a
> custom_target cannot have a dependency on a run_target. As also
> mentioned above, the whole meson implementation is a bit ugly.
>
> Anyway, with the attached patch you can do
>
> make dist
>
> or
>
> meson compile -C build pgdist

I played this with meson build on macOS, the packages are generated
in source root but not build root, I'm sure if this is by design but I think
polluting *working directory* is not good.

Another thing I'd like to point out is, should we also introduce *git commit*
or maybe *git tag* to package name, something like:

git archive --format tar.gz --prefix postgresql-17devel/ HEAD -o
postgresql-17devel-`git rev-parse --short HEAD`.tar.gz
git archive --format tar.gz --prefix postgresql-17devel/ HEAD -o
postgresql-`git describe --tags`.tar.gz

>
> and it produces the same set of tarballs as before, except it's done
> differently.
>
> The actual build scripts need some fine-tuning, but the general idea is
> correct, I think.

I think this is a good idea, thanks for working on this.

--
Regards
Junwang Zhao