make dist using git archive

From: Peter Eisentraut <peter(at)eisentraut(dot)org>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: make dist using git archive
Date: 2024-01-22 07:31:59
Message-ID: 40e80f77-a294-4f29-a16f-e21bc7bc75fc@eisentraut.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

One of the goals is to make the creation of the distribution tarball
more directly traceable to the git repository. That is why we removed
the "make distprep" step.

Here I want to take another step in that direction, by changing "make
dist" to directly use "git archive", rather than the custom shell script
it currently runs.

The simple summary is that it would run

git archive --format tar.gz --prefix postgresql-17devel/ HEAD -o
postgresql-17devel.tar.gz

(with appropriate version numbers, of course), and that's the tarball we
would ship.

There are analogous commands for other compression formats.

The actual command gets subtly more complicated if you need to run this
in a separate build directory. In my attached patch, the make version
doesn't support vpath at the moment, just so that it's easier to
understand for now. The meson version is a bit hairy.

I have studied and tested this quite a bit, and I have found that the
archives produced this way are deterministic and reproducible, meaning
for a given commit the result file should always be bit-for-bit identical.

The exception is that if you use a git version older than 2.38.0, gzip
records the platform in the archive, so you'd get a different output on
Windows vs. macOS vs. "UNIX" (everything else). In git 2.38.0, this was
changed so that everything is recorded as "UNIX" now. This is just
something to keep in mind. This issue is specific to the gzip format,
it does not affect other compression formats.

Meson has its own distribution building command (meson dist), but opted
against using that. The main problem is that the way they have
implemented it, it is not deterministic in the above sense. (Another
point is of course that we probably want a "make" version for the time
being.)

But the target name "dist" in meson is reserved for that reason, so I
needed to call the custom target "pgdist".

I did take one idea from meson: It runs a check before git archive that
the checkout is clean. That way, you avoid mistakes because of
uncommitted changes. This works well in my "make" implementation. In
the meson implementation, I had to find a workaround, because a
custom_target cannot have a dependency on a run_target. As also
mentioned above, the whole meson implementation is a bit ugly.

Anyway, with the attached patch you can do

make dist

or

meson compile -C build pgdist

and it produces the same set of tarballs as before, except it's done
differently.

The actual build scripts need some fine-tuning, but the general idea is
correct, I think.

Attachment Content-Type Size
v0-0001-make-dist-uses-git-archive.patch text/plain 3.2 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Michael Paquier 2024-01-22 07:36:27 Re: Use of backup_label not noted in log
Previous Message Masahiko Sawada 2024-01-22 07:23:43 Re: [PoC] Improve dead tuple storage for lazy vacuum