Re: make installcheck-world in a clean environment

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Fri, May 4, 2018 at 8:30 AM, Alexander Lakhin <exclusion(at)gmail(dot)com> wrote:
>> I'm not seeing a workaround to perform more complete installcheck without
>> modifying makefiles. So for me the question is whether the increasing the
>> testing surface justifies this use of time.

> After thinking about this some more, I think the question here is
> definitional. A first attempt at defining 'make installcheck' is to
> say that it runs the tests from the build tree against the running
> server. Certainly, we intend to use the SQL files that are present in
> the build tree, not the ones that were present in the build tree where
> the running server was built. But what about client programs that we
> use to connect to the server? You're suggesting that we use the
> pre-installed ones, but that is actually pretty problematic because
> the ones we see as installed might correspond neither to the contents
> of the build tree nor to the running server. Conceivably we could end
> up having a mix of assets from three different places: (1) the running
> server, (2) the build tree, (3) whatever is in our path at the moment.
> That seems very confusing. So now I think it's probably right to
> define 'make installcheck' as using the assets from the build tree to
> test the running server. Under that definition, we're missing some
> dependencies, but USE_INSTALLED_ASSETS isn't a thing we need.

Nah, I disagree with this. To me, the purpose of "make installcheck"
is to verify the correctness of an installation, which I take to include
the client programs as well as the server. I think that "make
installcheck" ought to use the installed version of any file that we
actually install, and go to the build tree only for things we don't
install (e.g. SQL test scripts).

If the user has screwed up his PATH or other environmental aspects so that
what he's testing isn't a single installation, that's his error, not
something that "make installcheck" ought to work around. Indeed, maybe
such aspects of his setup are intentional, and second-guessing them would
completely defeat his purpose. In any case, if you want to test the
build-tree assets, that's what "make check" is for.

regards, tom lane