Re: Improving tracking/processing of buildfarm test failures

On Thu, May 23, 2024 at 4:30 PM Alexander Lakhin <exclusion(at)gmail(dot)com> wrote:
>
> I'd like to discuss ways to improve the buildfarm experience for anyone who
> are interested in using information which buildfarm gives to us.
>
> Unless I'm missing something, as of now there are no means to determine
> whether some concrete failure is known/investigated or fixed, how
> frequently it occurs and so on... From my experience, it's not that
> unbelievable that some failure occurred two years ago and lost in time was
> an indication of e. g. a race condition still existing in the code/tests
> and thus worth fixing. But without classifying/marking failures it's hard
> to find such or other interesting failure among many others...
>
> The first way to improve things I can imagine is to add two fields to the
> buildfarm database: a link to the failure discussion (set when the failure
> is investigated/reproduced and reported in -bugs or -hackers) and a commit
> id/link (set when the failure is fixed). I understand that it requires
> modifying the buildfarm code, and adding some UI to update these fields,
> but it allows to add filters to see only unknown/non-investigated failures
> in the buildfarm web interface later.
>
> The second way is to create a wiki page, similar to "PostgreSQL 17 Open
> Items", say, "Known buildfarm test failures" and fill it like below:
> <url to failure1>
> <url to failure2>
> ...
> Useful info from the failure logs for reference
> ...
> <link to -hackers thread>
> ---
> This way is less invasive, but it would work well only if most of
> interested people know of it/use it.
> (I could start with the second approach, if you don't mind, and we'll see
> how it works.)
>

I feel it is a good idea to do something about this. It makes sense to
start with something simple and see how it works. I think this can
also help us whether we need to chase a particular BF failure
immediately after committing.

--
With Regards,
Amit Kapila.