Re: Improving tracking/processing of buildfarm test failures

Hello hackers,

Please take a look at the September report on buildfarm failures:
# SELECT br, count(*) FROM failures WHERE dt >= '2024-09-01' AND
 dt < '2024-10-01' GROUP BY br;
REL_12_STABLE: 12
REL_13_STABLE: 11
REL_14_STABLE: 12
REL_15_STABLE: 22
REL_16_STABLE: 35
REL_17_STABLE: 19
master: 33
-- Total: 144
(Counting test failures only, excluding indent-check, Configure, Build
errors.)

# SELECT COUNT(*) FROM (SELECT DISTINCT issue_link FROM failures WHERE
 dt >= '2024-09-01' AND dt < '2024-10-01');
16

# SELECT issue_link, count(*) FROM failures WHERE dt >= '2024-09-01' AND
 dt < '2024-10-01' GROUP BY issue_link ORDER BY 2 DESC LIMIT 6;
https://www.postgresql.org/message-id/ZuOKOut5hhDlf_bP@nathan: 51
-- Fixed

https://www.postgresql.org/message-id/20240924224352.93.nmisch@google.com: 31
-- Fixed

https://www.postgresql.org/message-id/a9a97e83-9ec8-5de5-bf69-80e9560f5345@gmail.com: 8
-- An environmental issue?; still not fixed

https://www.postgresql.org/message-id/657815a2-5a89-fcc1-1c9d-d77a6986bc26@gmail.com: 5

https://www.postgresql.org/message-id/CA+HiwqHZKNKOOcutAuzLmAWxEk-T+bgCMCT4nm6QEng0XRHBwQ@mail.gmail.com: 3
-- Fixed

https://www.postgresql.org/message-id/ZuOKOut5hhDlf_bP%40nathan: 3

# SELECT count(*) FROM failures WHERE dt >= '2024-09-01' AND
 dt < '2024-10-01' AND issue_link IS NULL; -- Unsorted/unhelpful failures
28

Short-lived failures: 91

Also please find attached the updated script I use to check the failures.

Best regards,
Alexander