Re: Buildfarm feature request: some way to track/classify failures

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> Martijn van Oosterhout wrote:
>> Maybe a simple compromise would be being able to setup a set of regexes
>> that search the output and set a flag it that string is found. If you
>> find the string, it gets marked with a flag, which means that when you
>> look at mongoose, any failures that don't have the flag become easier
>> to spot.
>>
>> It also means that once you've found a common failure, you can create
>> the regex and then any other failures with the same string get tagged
>> also, making unexplained ones easier to spot.

> You need to show first that this is an adequate tagging mechanism, both
> in tagging things adequately and in not picking up false positives,
> which would make things worse, not better. And even then you need
> someone to do the analysis to create the regex.

Well, my experiment over the weekend with doing exactly that convinced
me that regexes could be used successfully to identify common-mode
failures. So I think Martijn has a fine idea here. And I don't see a
problem with lack of motivation, at least for those of us who try to pay
attention to buildfarm results --- once you've looked at a couple of
reports of the same issue, you really don't want to have to repeat the
analysis over and over. But just assuming that every report on a
particular day reflects the same breakage is exactly the risk I wish
we didn't have to take.

For a lot of cases there is not a need for an ongoing filter: we break
something, we get a pile of reports, we fix it, and then we want to tag
all the reports of that something so that we can see if anything else
happened in the same interval. So for this, something based on an
interactive search API would work fine. You could even use that for
repetitive problems such as buildfarm misconfigurations, though having
to repeat the search every so often would get old in the end. The main
thing though is for the database to remember the tags once made.

> The buildfarm works because it leverages our strength, namely automating
> things. But all the tagging suggestions I've seen will involve regular,
> repetitive and possibly boring work, precisely the thing we are not good
> at as a group.

Well, responding to bug reports could be called regular and repetitive
work --- in reality I don't find it so, because every bug is different.
The point I think you are missing is that having something like this
will *eliminate* repetitive, boring work, namely recognizing multiple
reports of the same problem. The buildfarm has gotten big enough that
some way of dealing with that is desperately needed, else our ability
to spot infrequently-reported issues will disappear entirely.

regards, tom lane