Re: TAP tests - installcheck vs check

A couple of thoughts I've been having that relate to this:

The traditional meaning of "installcheck" in GNU packages is to test
against the installed code, whereas "check" tests before installation.
Our concept of testing against a running server obviously does not apply
to many kinds of software, so there is no standard name for it.

I think as our test suites expand, the concept of testing against a
running server is not going to feasible for many new and interesting
things. I can see some marginal appeal for what what we're currently
doing, but I don't think it's useful to invest much more effort into it
to make it work in more cases and situations.

What I'm thinking going forward is:

make check - as is

make installcheck - test against installed code, but start new instances
etc. (which you currently can't do for the main test suite)

make runningcheck - test against running server, supported for certain tests

make check and installcheck would be very similar except make check
creates a temp install and sets some path variables.

A trick of sorts I saw in Python packages that might also be worth
adopting is to make the temp install use symlinks pointing into the
source tree. Then as soon as you rebuild, the temp install is up to
date (unless you add or remove entire files). Then the temp install is
free after the first run.

Thoughts?

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services