Re: [Pgbuildfarm-members] Version 4.10 of buildfarm client released.

On 01/11/2013 01:39 PM, Tom Lane wrote:
> Heikki Linnakangas <hlinnaka(at)iki(dot)fi> writes:
>> There was a stray postgres instance running on the box, which I killed:
> FWIW, we've seen an awful lot of persistent buildfarm failures that
> seemed to be due to port conflicts with leftover postmasters. I think
> the buildfarm script needs to try harder to ensure that it's killed
> everything after a run. No good ideas how to go about that exactly.
> You could look through "ps" output for postmasters, but what if there's
> a regular Postgres installation on the same box? Can we just document
> that the buildfarm had better not be run as "postgres"? (If so, its
> attempt to kill an unowned postmaster would fail anyway; else we need
> a reliable way to tell which ones to kill.)
>
>

The buildfarm never builds with the standard port unless someone is
quite perverse indeed. The logic that governs it is:

$buildport = $PGBuild::conf{base_port};
if ($branch =~ /REL(\d+)_(\d+)/)
{
$buildport += (10 * ($1 - 7)) + $2;
}

Certainly the script should not be run as the standard postgres user.

Part of the trouble with detecting rogue postmasters it might have left
lying around is that various things like to decide what port to run on,
so it's not always easy for the buildfarm to know what it should be
looking for.

For branches >= 9.2 this is somewhat ameliorated by the existence of
EXTRA_REGRESS_OPTS, although we might need a slight adjustment to
pg_upgrade's test.sh to stop it from trampling on that willy-nilly.

I'm certainly reluctant to be trying to kill anything we aren't dead
certain is ours. We could possibly detect very early that there is a
suspected rogue postmaster.

One major source of these rogue processes has almost certainly been this
piece of logic in pg_ctl:

* The postmaster should create postmaster.pid very soon after being
* started. If it's not there after we've waited 5 or more seconds,
* assume startup failed and give up waiting.

WHen that happens, pg_ctl fails, and thus so does the buildfarmj client,
but if it has in fact started a postmaster that was just very slow in
writing its pid file it has left a postmastr lying around.

ISTR we discussed this phenomenon relatively recently, but I can't find
a reference to it readily. In any case, nothing has changed on that front.

cheers

andrew