Re: memory barriers (was: Yes, WaitLatch is vulnerable to weak-memory-ordering bugs)

On Thu, Sep 15, 2011 at 11:57 AM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> s/atomic/barrier/

Oops.

>> +/*
>> + * A compiler barrier need not (and preferably should not) emit any
>> actual
>> + * machine code, but must act as an optimization fence: the compiler must
>> not
>> + * reorder loads or stores to main memory around the barrier.  However,
>> the
>> + * CPU may still reorder loads or stores at runtime, if the
>> architecture's
>> + * memory model permits this.
>> + *
>> + * A memory barrier must act as a compiler barrier, and in addition must
>> + * guarantee that all loads and stores issued prior to the barrier are
>> + * completed before any loads or stores issued after the barrier.  Unless
>> + * loads and stores are totally ordered (which is not the case on most
>> + * architectures) this requires issuing some sort of memory fencing
>> + * instruction.
>
> This seems like a strange way to explain the problem. I would suggest
> structuring those paragraphs along the lines of:
>
> "
> A PostgreSQL memory barrier guarantees that any loads/stores before the
> barrier are completely finished and visible to other CPUs, before the
> loads/stores after the barrier are performed.
>
> That involves two things: 1. We must stop the compiler from rearranging
> loads/stores across the barrier. 2. We must stop the CPU from reordering the
> loads/stores across the barrier.
> "

That doesn't seem much different than I wrote?

One thing to keep in mind about whatever language we use here is that
memory barrier instructions need not (and often do not) compel the CPU
to "completely finish" anything before doing the next thing.
Execution is heavily pipelined, and on a sequence like STORE/WRITE
BARRIER/STORE the CPU is perfectly well entitled to begin the second
store before the first one is done. It just can't let the second
STORE get far enough for other CPUs to see it.

> Do we have any use for compiler barriers that are not also memory barriers?
> If not, I would suggest not exposing the pg_compiler_barrier() macro, but
> keep that as an implementation detail in the implementations of
> pg_memory_barrier().

I think there might be some use for a pure compiler barrier, but I'm
not sure yet. It's probably not worth having a "fallback"
implementation involving a spinlock, though, because odds are good
that any code that is performance-critical enough to be attempting to
safely use a compiler barrier can't survive having a spinlock
acquire-and-release shoved in there, so any such code should probably
be #ifdef'd.

> Some examples on the correct usage of these barriers would be nice, too.

That's a big topic. A trivial example is that if you write:

a[*x] = i;
++*x;

...where x and i are pointers into shared memory, another backend
might loop over a from 0 to x-1 and accidentally read off the end of
the array.

But even a full explanation of that case seems like almost too much
for the comment of a header file, and there are certainly far more
complex cases. I think anyone who is using these primitives is going
to have to do some independent reading...

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company