Re: tap tests driving the database via psql

On Wed, 31 Jul 2019 at 10:20, Andres Freund <andres(at)anarazel(dot)de> wrote:

> Hi,
>
> On 2019-07-31 09:32:10 +0800, Craig Ringer wrote:
> > OK. So rather than building our own $everything from scratch, lets look
> at
> > solving that.
>
> IDK, a minimal driver that just does what we need it to do is a few
> hundred lines, not more. And there's plenty of stuff that we simply
> won't be able to test with any driver that's not purposefully written
> for testing. There's e.g. basically no way with any of the drivers to
> test intentionally bogus sequences of protocol messages, yet that's
> something we really ought to test.
>
> I've written custom hacks^Wprograms to tests things like that a few
> times. That really shouldn't be necessary.
>
>
That's a good point. I've had to write simple protocol hacks myself, and
having a reusable base tool for it would indeed be valuable.

I'm just worried it'll snowball into Yet Another Driver We Don't Need, or
land up as a half-finished poorly-maintained burden nobody wants to touch.
Though the general fondness for and familiarity with Perl in the core
circle of Pg contributors and committers would probably help here, since
it'd inevitably land up being written in Perl...

I'm interested in evolution of the protocol and ways to enhance it, and I
can see something we can actively use for protocol testing being valuable.
If done right, the protocol implementation could probably pass-through
inspect messages from regular libpq etc as well as serving as either a
client or even as a dumb server simulator for pregenerated responses for
client testing.

We certainly couldn't do anything like that with existing tools and by
reusing existing drivers.

I wonder if there's any way to make writing and maintaining it less painful
though. Having to make everything work with Perl 5.8.8 and no non-core
modules leads to a whole pile of bespoke code and reinvention.

> > Perl modules support local installations. Would it be reasonable to
> require
> > DBI to be present, then rebuild DBD::Pg against our libpq during our own
> > test infra compilation, and run it with a suitable LD_LIBRARY_PATH or
> using
> > rpath? That'd actually get us a side-benefit of some test coverage of
> > DBD::Pg and libpq.
>
> You're intending to download DBD::Pg?

That'd be a reasonable option IMO, yes. Either via git with https, or a
tarball where we check a signature. So long as it can use any pre-existing
local copy without needing to hit the Internet, and the build can also
succeed without the presence of it at all, I think that'd be reasonable
enough.

But I'm probably getting contaminated by excessive exposure to Apache Maven
when I have to work with Java. I'm rapidly getting used to downloading
things being a part of builds. Personally I'd expect that unless
specifically testing new libpq functionality etc, most people would just be
using their system DBD::Pg... and I consider linking the system DBD::Pg
against our new-compiled libpq more feature than bug, as if we retain the
same soname we should be doing a reasonable job of not breaking upward
compatibility anyway. (It'd potentially be an issue if you have a very new
system libpq and are running tests on an old postgres, I guess).

> Or how would you get around the licensing issues? Downloading it will have
> some issues too: For one at least I
> often want to be able to run tests when offline; there's security
> concerns too.
>

Roughly what I was thinking of was:

For Pg source builds (git, or dist tarball), we'd generally curl a tarball
of DBD::Pg from a HTTPs URL specified in Makefile variables (with =? so it
can be overridden to point to an alt location, different version, local
path, etc) and unpack it, then build it. The makefile can also store a
signing key fingerprint so we can download the sig file and check the sig
is by the expected signer if the user has imported the signing key for
DBD::Pg.

We could test if the target directory already exists and is populated and
re-use it, so people can git-clone DBD::Pg if they prefer.

And we'd allow users to specify --with-system-dbd-pg at configure time, or
--without-dbd-pg .

The Pg perl libraries for our TAP test harness/wrappers would offer a
simple function to skip a test if DBD::Pg is missing, a simple function to
skip a test if the loaded DBD::Pg lacks $some_feature_or_attribute, etc.

>
> Greetings,
>
> Andres Freund
>

--
Craig Ringer http://www.2ndQuadrant.com/
2ndQuadrant - PostgreSQL Solutions for the Enterprise