Re: Recording test runtimes with the buildfarm

On 6/10/20 6:00 PM, Andres Freund wrote:
> On June 10, 2020 2:13:51 PM PDT, David Rowley <dgrowleyml(at)gmail(dot)com> wrote:
>>On Thu, 11 Jun 2020 at 02:13, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> I have in the past scraped the latter results and tried to make sense
>>of
>>> them. They are *mighty* noisy, even when considering just one animal
>>> that I know to be running on a machine with little else to do.
>>
>>Do you recall if you looked at the parallel results or the serially
>>executed ones?
>>
>>I imagine that the parallel ones will have much more noise since we
>>run the tests up to 20 at a time. I imagine probably none, or at most
>>not many of the animals have enough CPU cores not to be context
>>switching a lot during those the parallel runs. I thought the serial
>>ones would be better but didn't have an idea of they'd be good enough
>>to be useful.
>
> I'd assume that a rolling average (maybe 10 runs or so) would hide noise enough to see at least some trends even for parallel runs.
>
> We should be able to prototype this with a few queries over the bf database, right?

This seems to me like a perfect use case for control charts:

https://en.wikipedia.org/wiki/Control_chart

They are designed specifically to detect systematic changes in an environment
with random noise.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development