Re: CLOG extension

On Thu, May 3, 2012 at 1:27 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> Why not switch to 1 WAL record per file, rather than 1 per page. (32
> pages, IIRC).
>
> We can then have the whole new file written as zeroes by a background
> process, which needn't do that while holding the XidGenLock.

I thought about doing a single record covering a larger number of
pages, but that would be an even bigger hit if it were ever to occur
in the foreground path, so you'd want to be very sure that the
background process was going to absorb all the work. And if the
background process is going to absorb all the work, then I'm not sure
it matters very much whether we emit one xlog record or 32. After all
it's pretty low volume compared to all the other xlog traffic. Maybe
there's some room for optimization here, but it doesn't seem like the
first thing to pursue.

Doing it a background process, though, may make sense. What I'm a
little worried about is that - on a busy system - we've only got about
2 seconds to complete each CLOG extension, and we must do an fsync in
order to get there. And the fsync can easily take a good chunk of (or
even more than) that two seconds. So it's possible that saddling the
bgwriter with this responsibility would be putting too many eggs in
one basket. We might find that under the high-load scenarios where
this is supposed to help, bgwriter is already too busy doing other
things, and it doesn't get around to extending CLOG quickly enough.
Or, conversely, we might find that it does get around to extending
CLOG quickly enough, but consequently fails to carry out its regular
duties. We could of course add a NEW background process just for this
purpose, but it'd be nicer if we didn't have to go that far.

> My earlier patch to do background flushing from bgwriter can be
> extended to do that.

I've just been looking at that patch again, since as we discussed
before commit 3ae5133b1cf478d516666f2003bc68ba0edb84c7 fixed a problem
in this area, and it may be that we can now show a benefit of this
approach where we couldn't before. I think it's separate from what
we're discussing here, so let me write more about that on another
thread after I poke at it a little more.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company