Re: GDQ iimplementation

On Tue, 2010-05-18 at 01:53 +0200, Hannu Krosing wrote:
> On Mon, 2010-05-17 at 14:46 -0700, Josh Berkus wrote:
> > Jan, Marko, Simon,
> >
> > I'm concerned that doing anything about the write overhead issue was
> > discarded almost immediately in this discussion.
>
> Only thing we can do to write overhead _on_master_ is to trade it for
> transaction boundary reconstruction on slave (or special intermediate
> node), effectively implementing a "logical WAL" in addition to (or as an
> extension of) the current WAL.

That does sound pretty good to me.

Fairly easy to make the existing triggers write XLOG_NOOP WAL records
directly rather than writing to a queue table, which also gets logged to
WAL. We could just skip the queue table altogether.

Even better would be extending WAL format to include all the information
you need, so it gets written to WAL just once.

> > This is not a trivial
> > issue for performance; it means that each row which is being tracked by
> > the GDQ needs to be written to disk a minimum of 4 times (once to WAL,
> > once to table, once to WAL for queue, once to queue).
>
> In reality the WAL record for main table is forced to disk mosttimes in
> the same WAL write as the WAL record for queue. And the actual queue
> page does not reach disk at all if queue rotation is fast.

Josh, you really should do some measurements to show the overheads. Not
sure you'll get people just to accept that assertion otherwise.

--
Simon Riggs www.2ndQuadrant.com