Re: The xversion-upgrade test fails to stop server

Hello,

30.05.2024 18:00, Alexander Lakhin wrote:
> While reviewing recent buildfarm failures, I came across this one:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2024-05-23%2004%3A11%3A03
>
> upgrade.crake/REL_16_STABLE/REL9_5_STABLE-ctl4.log
> waiting for server to shut
> down...........................................................................................................................
> failed
> pg_ctl: server does not shut down
>

I've grepped through logs of the last 167
xversion-upgrade-REL9_5_STABLE-REL_16_STABLE/*ctl4.log on crake and got
the following results:
waiting for server to shut down........ done
waiting for server to shut down............................... done
waiting for server to shut down.............. done
waiting for server to shut down........ done
waiting for server to shut down....................... done
waiting for server to shut down..... done
waiting for server to shut down...... done
waiting for server to shut down............................... done
waiting for server to shut down.... done
waiting for server to shut down....................................... done
waiting for server to shut down....................... done
waiting for server to shut down............................................ done
waiting for server to shut down..... done
waiting for server to shut down....................... done
waiting for server to shut down.... done
waiting for server to shut down..... done
waiting for server to shut down................... done
waiting for server to shut down...................................... done
waiting for server to shut down........ done
waiting for server to shut down................ done
waiting for server to shut down............ done
waiting for server to shut down........... done
waiting for server to shut down........ done
waiting for server to shut down.... done
waiting for server to shut down.................................................. done
waiting for server to shut down........................................................... done
waiting for server to shut down..... done
waiting for server to shut down.............. done
waiting for server to shut down......... done
waiting for server to shut down........................................................... done
waiting for server to shut down................ done
waiting for server to shut down.................................... done
waiting for server to shut down........ done
waiting for server to shut down............ done
waiting for server to shut down................................................................... done
waiting for server to shut down..... done
waiting for server to shut down.............................................. done
waiting for server to shut down............. done
waiting for server to shut down............. done
waiting for server to shut down..... done
waiting for server to shut down.............................................. done
waiting for server to shut down...... done
waiting for server to shut down....... done
waiting for server to shut down....... done
waiting for server to shut down.......................... done
waiting for server to shut down............ done
waiting for server to shut down.................... done
waiting for server to shut down..... done
waiting for server to shut down........ done
waiting for server to shut down....... done
waiting for server to shut down....................... done
waiting for server to shut down........... done
waiting for server to shut down.............. done
waiting for server to shut down.................... done
waiting for server to shut down............. done
waiting for server to shut down...................................... done
waiting for server to shut down............................................... done
waiting for server to shut down........................................ done
waiting for server to shut down.......... done
waiting for server to shut down......................................... done
waiting for server to shut down......................................... done
waiting for server to shut down........ done
waiting for server to shut down..... done
waiting for server to shut down................................................................... done
waiting for server to shut down............................. done
waiting for server to shut down.......... done
waiting for server to shut down..... done
waiting for server to shut down......................... done
waiting for server to shut down...... done
waiting for server to shut down..... done
waiting for server to shut down......... done
waiting for server to shut down.......................................................... done
waiting for server to shut down............. done
waiting for server to shut down..... done
waiting for server to shut down........ done
waiting for server to shut down........................................... done
waiting for server to shut down.... done
waiting for server to shut down..... done
waiting for server to shut down...... done
waiting for server to shut down............. done
waiting for server to shut down.......................................................... done
waiting for server to shut down........................... done
waiting for server to shut down..... done
waiting for server to shut down....... done
waiting for server to shut down...... done
waiting for server to shut down......................................... done
waiting for server to shut down.......................... done
waiting for server to shut down.......................... done
waiting for server to shut down................................... done
waiting for server to shut down..... done
waiting for server to shut down.................. done
waiting for server to shut down....................... done
waiting for server to shut down................. done
waiting for server to shut down........................................... done
waiting for server to shut down........ done
waiting for server to shut down...................................................................... done
waiting for server to shut down............. done
waiting for server to shut down.................................................................... done
waiting for server to shut down................... done
waiting for server to shut down.............. done
waiting for server to shut down..................... done
waiting for server to shut
down.................................................................................................... done
waiting for server to shut down..... done
waiting for server to shut down....................... done
waiting for server to shut down................................ done
waiting for server to shut down....... done
waiting for server to shut down....... done
waiting for server to shut down.............. done
waiting for server to shut down................................................ done
waiting for server to shut down...... done
waiting for server to shut down..................... done
waiting for server to shut down...... done
waiting for server to shut down........ done
waiting for server to shut down..... done
waiting for server to shut down.... done
waiting for server to shut down......... done
waiting for server to shut down...... done
waiting for server to shut down........... done
waiting for server to shut down...................... done
waiting for server to shut down............ done
waiting for server to shut down................. done
waiting for server to shut down.... done
waiting for server to shut down................................... done
waiting for server to shut down.............................. done
waiting for server to shut down....................................................................................... done
waiting for server to shut
down................................................................................................... done
waiting for server to shut down...... done
waiting for server to shut down.... done
waiting for server to shut down........... done
waiting for server to shut down..... done
waiting for server to shut down......... done
waiting for server to shut down.................... done
waiting for server to shut down..... done
waiting for server to shut down....................... done
waiting for server to shut down................. done
waiting for server to shut down........................................ done
waiting for server to shut down...... done
waiting for server to shut down.... done
waiting for server to shut down.................................... done
waiting for server to shut down.............................. done
waiting for server to shut down.................................. done
waiting for server to shut down.............................................................. done
waiting for server to shut down.................................................. done
waiting for server to shut down............................................................. done
waiting for server to shut down............ done
waiting for server to shut down............ done
waiting for server to shut down...................................... done
waiting for server to shut down.......................................... done
waiting for server to shut down................................. done
waiting for server to shut down................... done
waiting for server to shut
down...........................................................................................................................
failed
waiting for server to shut down................................................. done
waiting for server to shut down............ done
waiting for server to shut down................................................................... done
waiting for server to shut down......... done
waiting for server to shut down......... done
waiting for server to shut down.................... done
waiting for server to shut down........ done
waiting for server to shut down............................... done
waiting for server to shut down......... done
waiting for server to shut down......................................................... done
waiting for server to shut down................................................... done
waiting for server to shut down............ done
waiting for server to shut down...... done
waiting for server to shut down........................................... done
waiting for server to shut
down.......................................................................................... done
waiting for server to shut down............................................................................... done

Thus, pg_ctl stopped waiting after 120 seconds timeout, but we can see
"allowed" duration around 100 seconds.

A similar failure have occurred today:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=opaleye&dt=2024-06-08%2001%3A41%3A41
waiting for server to shut
down...........................................................................................................................
failed
pg_ctl: server does not shut down

and the last StopDb-C:4 runs on opaleye show the following timings:
waiting for server to shut down............. done
waiting for server to shut down......................... done
waiting for server to shut down......................... done
waiting for server to shut down.......................... done
waiting for server to shut down.............. done
waiting for server to shut down...................... done
waiting for server to shut down................................. done
waiting for server to shut down........................... done
waiting for server to shut down............ done
waiting for server to shut down................ done
waiting for server to shut down.................. done
waiting for server to shut down............................... done
waiting for server to shut down............................ done
waiting for server to shut down.................. done
waiting for server to shut down..................... done
waiting for server to shut down........................ done
waiting for server to shut down.......................... done
waiting for server to shut down......................................... done
waiting for server to shut down.................... done
waiting for server to shut down..................................... done
waiting for server to shut down............................................. done
waiting for server to shut down........................... done
waiting for server to shut down................... done
waiting for server to shut down............. done
waiting for server to shut down........................ done
waiting for server to shut down......................... done
waiting for server to shut down................... done
waiting for server to shut down................. done
waiting for server to shut down..................... done
waiting for server to shut down......................... done
waiting for server to shut down............................... done
waiting for server to shut down.................................................. done
waiting for server to shut down............... done
waiting for server to shut down................. done
waiting for server to shut down..................... done
waiting for server to shut down................ done
waiting for server to shut down.............. done
waiting for server to shut down................ done
waiting for server to shut down.......................... done
waiting for server to shut down.................. done
waiting for server to shut down.................... done
waiting for server to shut down................ done
waiting for server to shut down....................... done
waiting for server to shut down................ done
waiting for server to shut down...................... done
waiting for server to shut down............... done
waiting for server to shut down.............. done
waiting for server to shut down........................... done
waiting for server to shut down............. done
waiting for server to shut down..................... done
waiting for server to shut down................. done
waiting for server to shut down........................... done
waiting for server to shut down..................... done
waiting for server to shut down................ done
waiting for server to shut down........................................... done
waiting for server to shut down....................... done
waiting for server to shut down.............................................................................. done
waiting for server to shut
down...........................................................................................................................
failed

So maybe it would make sense to increase default PGCTLTIMEOUT for
buildfarm clients, say, to 180 seconds?

Best regards,
Alexander