Re: TAP backpatching policy

Craig,

* Craig Ringer (craig(at)2ndquadrant(dot)com) wrote:
> I propose that we backpatch all practical TAP enhancements back to the
> last supported back branch.

Thanks for bringing this up. I tend to agree, as we really want to have
better testing of PostgreSQL in all of our branches and being able to
back-patch tests, and add more tests to all of the branches where those
tests apply, will help a great deal with that.

> At the moment that'd be 9.5, since that's where PostgresNode was
> introduced. But if I can find the time I'd quite like to backport
> PostgresNode to 9.4 too.

Makes sense to me.

> Where needed, PostgresNode could have version-dependent logic so we
> could just copy the Perl module to the different branches.

Agreed.

> This would make it much easier to test for bugs when doing work on
> back branches, run the same test on multiple branches, etc. My
> personal motivation is mainly using it in extensions, but I've
> _frequently_ found myself writing TAP tests when looking into possible
> Pg bugs etc too. Not having the same facilities in the different
> branches makes this way harder.

I also find it really unfortunate to write a set of tests but then not
be able to have them included and run across the buildfarm. As much as
I'd like to think that testing locally identifies all issues, I've been
proven wrong enough to realize that it's really unlikely to ever be
true.

> If combined with pg_config --version-num (which IMO deserves a
> backpatch especially now Pg 10 makes parsing our versions harder) this
> would make it a lot more practical to do nontrivial tests in
> extensions - which really matters since we introduced bgworkers.

Presumably you mean nontrivial tests of extensions in *back-branches*,
here? Or am I misunderstanding what you're getting at?

> Thoughts? Backpatch new TAP methods, etc, into back branches routinely?

For my 2c, at least, I'd like to see the project, in general, be much
more accepting of adding tests to all branches, and, really, to
encourage that the focus of the time between feature-freeze and release
be specifically on improving our testing code coverage, including the
coverage in back branches, as well as the to-be-released version.

Perhaps, one day in the far future, we will be able to look back in
amusement at the lack of code coverage we have these days, but we have
to really work to get there and that means setting aside some period of
dedicated time during the year for writing tests, in my opinion, and not
being overly picky about if those tests are adding code coverage for
code committed in Postgres95 or for PG10. I would have thought that the
extent of the back-patching I've been doing of late for issues found in
pg_dump, back to the initial implementation of those parts, would point
out this need, but I'm afraid that there continues to be a general
feeling that "what we have is good enough" and I find that a very hard
pill to swallow when there are extremely important components of the
system which are, essentially, entirely untested in the buildfarm, such
as:

https://coverage.postgresql.org/src/backend/access/brin/brin_xlog.c.gcov.html
https://coverage.postgresql.org/src/backend/access/common/bufmask.c.gcov.html
https://coverage.postgresql.org/src/backend/access/gin/ginxlog.c.gcov.html
https://coverage.postgresql.org/src/backend/access/gist/gistbuildbuffers.c.gcov.html
https://coverage.postgresql.org/src/backend/access/gist/gistxlog.c.gcov.html
https://coverage.postgresql.org/src/backend/access/hash/hash_xlog.c.gcov.html
https://coverage.postgresql.org/src/backend/access/nbtree/nbtxlog.c.gcov.html
https://coverage.postgresql.org/src/backend/access/spgist/spgxlog.c.gcov.html
https://coverage.postgresql.org/src/backend/access/transam/xlogfuncs.c.gcov.html
https://coverage.postgresql.org/src/backend/executor/nodeCustom.c.gcov.html
https://coverage.postgresql.org/src/backend/executor/tqueue.c.gcov.html

Let's. please. work together to correct this.

Thanks!

Stephen