Re: Rewriting the test of pg_upgrade as a TAP test

Andres,

* Andres Freund (andres(at)anarazel(dot)de) wrote:
> On 2017-04-05 10:40:41 -0400, Stephen Frost wrote:
> > * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> > > Stephen Frost <sfrost(at)snowman(dot)net> writes:
> > > > I believe that what Peter was getting at is that the pg_dump TAP tests
> > > > create a whole slew of objects in just a few seconds and are able to
> > > > then exercise those code-paths in pg_dump, without needing to run the
> > > > entire serial regression test run.
> > >
> > > Right. But there's a certain amount of serendipity involved in using the
> > > core regression tests' final results. For example, I don't know how long
> > > it would've taken us to understand the problems around dumping and
> > > reloading child tables with inconsistent column orders, had there not been
> > > examples of that in the regression tests. I worry that creating a sterile
> > > set of objects for testing pg_dump will leave blind spots, because it will
> > > mean that we only test cases that we explicitly created test cases for.
> >
> > We don't need to only create sterile sets of objects in the pg_dump TAP
> > tests.
>
> I really, really don't understand why we're conflating making pg_upgrade
> tests less fragile / duplicative with changing what we use to test it.
> This seems to have the sole result that we're not going to get anywhere.

Probably because the point was brought up that the regression tests for
pg_upgrade spend a bunch of time doing something which, ultimately,
don't actually add any real value. Yes, there are bits of the core
regression tests that currently add value over what we have through
other approaches, but that's not where the bulk of running those tests
go.

> > I don't believe we need to populate GIN indexes or vacuum them
> > to test pg_dump/pg_upgrade either (at least, not if we're going to stick
> > to the pg_upgrade test basically being if pg_dump returns the same
> > results before-and-after).
>
> I think we *should* have populated GIN indexes. Yes, the coverage isn't
> perfect, but the VACUUM definitely gives a decent amount of coverage
> whether the gin index looks halfway sane after the upgrade.

We don't look at the gin index after the upgrade in the current
pg_upgrade testing, so I don't see why you feel it's at all valuable.
If we *did* do that (and I'm all for adding such tests), then perhaps
this argument would make sense, but we don't today and I haven't seen
anyone propose changing that.

Thanks!

Stephen