Re: FlexLocks

On Fri, Nov 18, 2011 at 6:26 AM, Pavan Deolasee
<pavan(dot)deolasee(at)gmail(dot)com> wrote:
> My apologies for hijacking the thread, but the work seems quite
> related, so I thought I should post here instead of starting a new
> thread.
>
> Here is a WIP patch based on the idea of having a shared Q. A process
> trying to access the shared memory protected by a LWLock, sets up the
> task in its PGPROC and calls a new API LWLockExecute(). If the LWLock
> is available, the task is performed immediately and the function
> returns. Otherwise, the process queues up itself on the lock. When the
> last shared lock holder or the exclusive lock holder call
> LWLockRelease(), it scans through such pending tasks, executes them
> via a callback mechanism and wakes all those processes along with any
> other normal waiter(s) waiting on LWLockAcquire().
>
> I have only coded for ProcArrayEndTransaction, but it should fairly
> easy to extend the usage at some more places, especially those which
> does some simple modifications to the protected area. I don't propose
> to use the technique for every user of LWLock, but there can be some
> obvious candidates, including this one that Robert found out.
>
> I see 35-40% improvement for 32-80 clients on a 5 minutes pgbench -N
> run with scale factor of 100 and permanent tables. This is on a
> 32-core HP IA box.
>
> There are few things that need some deliberations. The pending tasks
> are right now executed while holding the mutex (spinlock). This is
> good and bad for obvious reasons. We can possibly change that so that
> the work is done without holding the spinlock or leave to the caller
> to choose the behavior. Doing it without holding the spinlock will
> make the technique interesting for many more callers. We can also
> rework the task execution so that pending similar requests from
> multiple callers can be combined and executed with a single callback,
> if the caller knows its safe to do so. I haven't thought through the
> API/callback changes to support that, but its definitely possible and
> could be quite useful in many cases. For example, status of many
> transactions can be checked with a single lookup of the ProcArray. Or
> WAL inserts from multiple processes can be combined and written at
> once.

So the upside and downside of this approach is that it modifies the
existing LWLock implementation rather than allowing multiple lock
implementations to exist side-by-side. That means every LWLock in the
system has access to this functionality, which might be convenient if
there turn out to be many uses for this technique. The bad news is
that everyone pays the cost of checking the work queue in
LWLockRelease(). It also means that you can't, for example, create a
custom lock with different lock modes (e.g. S, SX, X, as I proposed
upthread).

I am pretty dubious that there are going to be very many cases where
we can get away with holding the spinlock while doing the work. For
example, WAL flush is a clear example of where we can optimize away
spinlock acquisitions - if we communicate to people we wake up that
their LSN is already flushed, they needn't reacquire the lock to
check. But we certainly can't hold a spinlock across a WAL flush.
The nice thing about the FlexLock approach is that it permits
fine-grained control over these types of policies: one lock type can
switch to exclusive mode, do the work, and then reacquire the spinlock
to hand off the baton; another can do the work while holding the
spinlock; and still a third can forget about work queues altogether
but introduce additional lock modes.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company