Re: Improving tracking/processing of buildfarm test failures

Hello hackers,

25.05.2024 15:00, I wrote:
> I've created such page to accumulate information on test failures:
> https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures
>

One month later,  I'd like to summarize failures that I've investigated
and classified during June, 2024 on the aforementioned wiki page.
(Maybe it would make sense to issue a monthly report with such information
in the future.)

Imagining a hypothetical table, we could get such statistics:
# SELECT br, count(*) FROM failures WHERE dt >= '2024-06-01' AND
 dt < '2024-07-01' GROUP BY br;
REL_12_STABLE: 6
REL_13_STABLE: 14
REL_14_STABLE: 13
REL_15_STABLE: 10
REL_16_STABLE: 4
HEAD: 47
-- Total: 94
(Counting test failures only, excluding indent-check, Configure, Build
errors.)

# SELECT COUNT(*) FROM (SELECT DISTINCT issue_link FROM failures WHERE
 dt >= '2024-06-01' AND dt < '2024-07-01');
21

# SELECT issue_link, count(*) FROM failures WHERE dt >= '2024-06-01' AND
 dt < '2024-07-01' GROUP BY issue_link ORDER BY 2 DESC LIMIT 7;
https://www.postgresql.org/message-id/20240628051353.a0.nmisch@google.com: 13
--
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#inplace-inval.spec_fails_on_prion_and_trilobite_on_checking_relhasindex
-- Fixed

https://www.postgresql.org/message-id/95ca84ca-39b4-f6aa-260f-da5f73d05a90@gmail.com: 10
--
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#008_fsm_truncation_failing_on_dodo_in_v14-_due_to_slow_fsync
-- An environmental issue

https://www.postgresql.org/message-id/f748ee55-9e73-3f5e-e879-8865c5e9933a@gmail.com: 9
-- https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#regress-running.2Fregress_fails_on_skink_due_to_timeout
-- An environmental issue

https://www.postgresql.org/message-id/d6ee8761-39d1-0033-1afb-d5a57ee056f2@gmail.com: 9
--
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#ssl_tests_.28001_ssltests.pl.2C_002_scram.pl.2C_003_sslinfo.pl.29_fail_due_to_TCP_port_conflict
-- A fix proposed, commit pending

https://www.postgresql.org/message-id/4cc2ee93-e03c-8e13-61ed-412e7e6ff19d@gmail.com: 9
--
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#plperl.sql_failing_in_v15-_on_caiman_with_a_newer_Perl_version
-- Fixed

https://www.postgresql.org/message-id/2509767.1719773880@sss.pgh.pa.us: 7
--
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#040_pg_createsubscriber.pl_fails_on_Windows_due_to_unterminated_quoted_string
-- Fixed

https://www.postgresql.org/message-id/847814.1715631450@sss.pgh.pa.us: 6
--
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#Isolation_tests_fail_on_hamerkop_with_.22too_many_clients.22_errors
-- A fix proposed, commit pending

# SELECT fix_link, count(*) FROM failures WHERE dt >= '2024-06-01' AND
 dt < '2024-07-01' AND fix_link IS NOT NULL GROUP BY fix_link ORDER BY 2 DESC;
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=458fada72: 13
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=f853e23bf: 10
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a1333ec04: 7
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=b96391382: 3
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=e656657f2: 1
-- Total: 5

# SELECT log_link FROM failures WHERE dt >= '2024-06-01' AND
 dt < '2024-07-01' AND issue_link IS NULL; -- Not investigated/classified failures

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=dodo&dt=2024-06-17%2004%3A21%3A42
initdb: error: invalid locale settings; check LANG and LC_* environment variables

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=kingsnake&dt=2024-06-27%2015%3A38%3A27
StopDb-C:4
pg_ctl: server does not shut down
-- The most mysterious issue to me, more information needed

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=kingsnake&dt=2024-06-13%2017%3A58%3A28
StopDb-C:4
pg_ctl: server does not shut down
-- The most mysterious issue to me, more information needed

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-06-28%2001%3A06%3A00
# Running: pg_ctl -D
C:\\prog\\bf\\root\\REL_16_STABLE\\pgsql.build/testrun/recovery/002_archiving\\data/t_002_archiving_standby_data/pgdata
-l C:\\prog\\bf\\root\\REL_16_STABLE\\pgsql.build/testrun/recovery/002_archiving\\log/002_archiving_standby.log promote
waiting for server to
promote...........................................................................................................................
stopped waiting
pg_ctl: server did not promote in time
-- Most probably the machine's performance issue, an issue report is pending.

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2024-06-04%2022%3A45%3A00
connection error: 'psql: error: connection to server on socket "/tmp/9IDPzZm7Pp/.s.PGSQL.63572" failed: FATAL:  role
"bf" does not exist'

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2024-06-17%2016%3A02%3A03&stg=xversion-upgrade-REL_16_STABLE-HEAD
program "postgres" is needed by pg_ctl but was not found in the same directory as
"/home/andrew/bf/root/saves.crake/REL_16_STABLE/bin/pg_ctl"

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2024-06-17%2017%3A07%3A03&stg=xversion-upgrade-REL_16_STABLE-HEAD
program "postgres" is needed by pg_ctl but was not found in the same directory as
"/home/andrew/bf/root/saves.crake/REL_16_STABLE/bin/pg_ctl"

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=caiman&dt=2024-07-01%2003%3A13%3A26
+ERROR:  could not access file "/repos/build-farm-17/HEAD/inst/lib/postgresql/plpgsql.so": No such file or directory

-- Total: 8

All the queries above are imaginary and some numbers could be inaccurate,
but I think it still represents the current state of affairs.

Best regards,
Alexander