Re: Improving tracking/processing of buildfarm test failures

Hello hackers,

Please take a look at the August report on buildfarm failures:
# SELECT br, count(*) FROM failures WHERE dt >= '2024-08-01' AND
 dt < '2024-09-01' GROUP BY br;
REL_12_STABLE: 2
REL_13_STABLE: 2
REL_14_STABLE: 12
REL_15_STABLE: 3
REL_16_STABLE: 5
REL_17_STABLE: 17
HEAD: 38
-- Total: 79
(Counting test failures only, excluding indent-check, Configure, Build
errors.)

# SELECT COUNT(*) FROM (SELECT DISTINCT issue_link FROM failures WHERE
 dt >= '2024-08-01' AND dt < '2024-09-01');
21

# SELECT issue_link, count(*) FROM failures WHERE dt >= '2024-08-01' AND
 dt < '2024-09-01' GROUP BY issue_link ORDER BY 2 DESC LIMIT 6;
https://www.postgresql.org/message-id/8ce8261a-bf3a-25e6-b473-4808f50a6ea7%40gmail.com: 13
-- An environmental issue; fixed

https://www.postgresql.org/message-id/a9a97e83-9ec8-5de5-bf69-80e9560f5345@gmail.com: 9
-- An environmental issue?; probably fixed

https://www.postgresql.org/message-id/4db099c8-4a52-3cc4-e970-14539a319466@gmail.com: 7
-- Fixed

https://www.postgresql.org/message-id/c720cdc3-5ce0-c410-4854-70788175ca2c@gmail.com: 6
-- Expected to be fixed with Release 18 of the buildfarm client

https://www.postgresql.org/message-id/657815a2-5a89-fcc1-1c9d-d77a6986bc26@gmail.com: 5

https://www.postgresql.org/message-id/3618203.1722473994@sss.pgh.pa.us: 4
-- Fixed

# SELECT count(*) FROM failures WHERE dt >= '2024-08-01' AND
 dt < '2024-09-01' AND issue_link IS NULL; -- Unsorted/unhelpful failures
13

Short-lived failures: 21

There were also two mysterious never-before-seen failures, both occurred on
POWER animals:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=kingsnake&dt=2024-08-19%2019%3A17%3A59 - REL_17_STABLE
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=iguana&dt=2024-08-29%2013%3A57%3A57 - REL_15_STABLE

(I'm not sure yet, whether they should be considered "unhelpful". I'll
wait for more information from these animals/buildfarm in general to
determine what to do with these failures.)

Best regards,
Alexander