Re: src/test/recovery regression failure on bionic

I wrote:
> This would happen if anything is causing the postmaster to have
> a few more open files than the test added by commit
> d207038053837ae9365df2776371632387f6f655 is allowing for. It's
> a test bug and nothing more.
> Why sidewinder is not showing this in HEAD too is an interesting
> question, but it isn't. However, it could be that on another
> platform (ie bionic) the problem does manifest in HEAD.

I set up a NetBSD 7 installation locally, and while I have not
directly reproduced the failure, I believe I understand all the
components of it now.

(1) d20703805's test will clearly fall over if there are more than six
FDs open in the postmaster when set_max_safe_fds is called, because it
sets max_files_per_process = 26 while set_max_safe_fds requires at
least 20 usable FDs to be available.

(2) The postmaster's stdin/stdout/stderr will surely eat up three of
those.

(3) In HEAD, that's actually all the FDs there are normally, but in the
back branches there is one more (under the conditions of this test),
because in the back branches we open the postmaster's listen sockets
before we run set_max_safe_fds. (9a86f03b4 changed this.)

(4) NetBSD 7.0's cron leaves three extra open FDs in processes that
it spawns. I have not looked into why, but I have experimentally
observed this. For example, lsof on a "sleep" launched from cron
shows

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sleep 7824 tgl cwd VDIR 0,0 512 795201 /home/tgl
sleep 7824 tgl txt VREG 0,0 10431 1613152 /bin/sleep
sleep 7824 tgl txt VREG 0,0 1616564 22726 /lib/libc.so.12.193.1
sleep 7824 tgl txt VREG 0,0 55295 22747 /lib/libgcc_s.so.1.0
sleep 7824 tgl txt VREG 0,0 187183 22762 /lib/libm.so.0.11
sleep 7824 tgl txt VREG 0,0 92195 1499524 /libexec/ld.elf_so
sleep 7824 tgl 0r PIPE 0xfffffe803131eb58 16384
sleep 7824 tgl 1w PIPE 0xfffffe8007ec4a30 0 ->0xfffffe800cc0d2c0
sleep 7824 tgl 2w PIPE 0xfffffe8007ec4a30 0 ->0xfffffe800cc0d2c0
sleep 7824 tgl 7u unknown file system type: 0
sleep 7824 tgl 8u unknown file system type: 0
sleep 7824 tgl 9w PIPE 0xfffffe80036c4dc0 0

while of course "sleep" launched by hand has only 0/1/2 open.

We may conclude that when the regression tests are launched from cron,
as would be typical for a buildfarm animal, HEAD has exactly zero FDs
to spare in this test, while the back branches are one FD underwater
and fail. This matches the observed results from sidewinder.

It's not clear whether any of this info applies to Christoph's trouble
with bionic. If the extra FDs are an old cron bug, it could be that
bionic shares that bug --- but to explain failure on HEAD, you'd have to
posit four excess FDs not three. I'm not convinced that what Christoph
is seeing matches this anyway; he hasn't showed the telltale
"insufficient file descriptors" message, at least. Still, maybe
launched-by-cron vs launched-by-hand is a relevant point there.

regards, tom lane