Re: Buildfarm's cross-version-upgrade tests

On Wed, Dec 30, 2020 at 04:28:44PM -0500, Andrew Dunstan wrote:
> On 12/30/20 3:42 PM, Tom Lane wrote:
> > Since I'd just seen Noah's commit 52202bb39 go by, I tried
> > modifying src/bin/pg_upgrade/test.sh to do the drop, but that
> > isn't helping. I recall now from prior discussions that we
> > have to modify code that's embedded in the buildfarm client
> > script to make this go. (I wonder what test scenario Noah had
> > in mind, exactly.)

Commit 52202bb39 affected the cross-version test harness described in
src/bin/pg_upgrade/TESTING. That harness is disused, having been broken at
head for at least 40 months. (One can work around the remaining breakage by
locally back-patching e78900a into REL_13_STABLE.)

> > I'm now wondering if we can't devise a fixup method that is
> > controllable by committers and doesn't require a buildfarm
> > client rollout to update.

Ideally, the buildfarm code should curate per-version source and installation
trees, provide their locations to the test harness, and collect logs. Other
test logic (i.e. the kinds of steps that appear in test.sh) should live in
postgresql.git code. (That's the separation of concerns for all
run-by-default buildfarm tests.)