Re: buffer assertion tripping under repeat pgbench load

On Thu, Dec 27, 2012 at 11:33 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Greg Stark <stark(at)mit(dot)edu> writes:
>> On Thu, Dec 27, 2012 at 3:17 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> The thing that this theory has a hard time with is that the buffer's
>>> global refcount is zero. If you assume that there's a bit that
>>> sometimes randomly goes to 1 when it should be 0, then what I'd expect
>>> to typically happen is that UnpinBuffer sees nonzero LocalRefCount and
>>> hence doesn't drop the session's global pin when it should. The only
>>> way that doesn't happen is if decrementing LocalRefCount to zero stores
>>> a nonzero pattern when it should store zero, but nonetheless the CPU
>>> thinks it stored zero.
>
>> It seems entirely plausible when you factor in the L2 cache. The cpu
>> could be happily incrementing and decrementing the count entirely
>> correctly blissfully unaware that the value being stored in the DRAM
>> has this extra bit set every time. Not until the transaction ends and
>> it has to refetch the cache line because enough time has passed for it
>> to age out of L2 cache does it find the corrupt value.
>
> Hmm ... that could be plausible. It would be good if we could reproduce
> this (or fail to) on some other machine. Greg mentioned running some
> memory diagnostics as well.

For what it's worth, I ran a bunch of pgbench runs at the same scale
factors, same settings for shared_buffers and checkpoint_segments, and
same scale factors that Greg mentioned. I used the IBM POWER7 box. I
did not get a failure there; but there are enough other variables that
I'm not sure this proves much.

I'm always skeptical of the explanation that the hardware is bad.
That could be it, but bad memory rarely causes such predictable
failures, at least IME.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company