Re: What in the world is happening with castoroides and protosciurus?

On Sat, Aug 30, 2014 at 11:32 PM, Noah Misch <noah(at)leadboat(dot)com> wrote:
> On Tue, Aug 26, 2014 at 10:17:05AM +0100, Dave Page wrote:
>> On Tue, Aug 26, 2014 at 1:46 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> > For the last month or so, these two buildfarm animals (which I believe are
>> > the same physical machine) have been erratically failing with errors that
>> > reflect low-order differences in floating-point calculations.
>> >
>> > A recent example is at
>> >
>> > http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=protosciurus&dt=2014-08-25%2010%3A39%3A52
>> >
>> > where the only regression diff is
>> >
>> > *** /export/home/dpage/pgbuildfarm/protosciurus/HEAD/pgsql.22860/src/test/regress/expected/hash_index.out Mon Aug 25 11:41:00 2014
>> > --- /export/home/dpage/pgbuildfarm/protosciurus/HEAD/pgsql.22860/src/test/regress/results/hash_index.out Mon Aug 25 11:57:26 2014
>> > ***************
>> > *** 171,179 ****
>> > SELECT h.seqno AS i8096, h.random AS f1234_1234
>> > FROM hash_f8_heap h
>> > WHERE h.random = '-1234.1234'::float8;
>> > ! i8096 | f1234_1234
>> > ! -------+------------
>> > ! 8906 | -1234.1234
>> > (1 row)
>> >
>> > UPDATE hash_f8_heap
>> > --- 171,179 ----
>> > SELECT h.seqno AS i8096, h.random AS f1234_1234
>> > FROM hash_f8_heap h
>> > WHERE h.random = '-1234.1234'::float8;
>> > ! i8096 | f1234_1234
>> > ! -------+-------------------
>> > ! 8906 | -1234.12356777216
>> > (1 row)
>> >
>> > UPDATE hash_f8_heap
>> >
>> > ... a result that certainly makes no sense. The results are not
>> > repeatable, failing in equally odd ways in different tests on different
>> > runs. This is happening in all the back branches too, not just HEAD.
>
>> I have
>> no idea what is causing the current issue - the machine is stable
>> software-wise, and only has private builds of dependency libraries
>> update periodically (which are not used for the buildfarm). If I had
>> to hazard a guess, I'd suggest this is an early symptom of an old
>> machine which is starting to give up.
>
> Agreed. Rerunning each animal against older commits would test that theory.
> Say, run against the last 6 months of REL9_0_STABLE commits. If those runs
> show today's failure frequencies instead of historic failure frequencies, it's
> not a PostgreSQL regression. Not that I see a commit back-patched near the
> time of the failure uptick (2014-08-06) that looks remotely likely to have
> introduced such a regression.
>
> It would be sad to lose our only buildfarm coverage of plain Solaris and of
> the Sun Studio compiler, but having buildfarm members this unstable is a pain.
> Perhaps have those animals retry the unreliable steps up to, say, 7 times?

That would require changes to the buildfarm client. I'll see if I can
find some alternate resources we can use.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company