Re: bgwriter and checkpoints

On Fri, Aug 12, 2011 at 2:19 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Aug 12, 2011 at 7:09 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> The bgwriter has been responsible for two main activities: incremental
>> page cleaning and checkpointing.
>>
>> We've worked out the code to smooth checkpointing, but that now means
>> we have periods where we do both page cleaning and checkpointing, and
>> other times when we do just page cleaning. That means the processing
>> loops are complex and that makes latch based processing more difficult
>> to introduce.
>
> I've thought about this before, and I think it's probably a good idea.
>  At the very least that code needs some refactoring, and breaking it
> into two completely separate processes may be the way to go.
>
> One possible trouble spot is that this will have the effect of
> increasing traffic on BgWriterCommLock.  Right now, page-cleaning
> activity doesn't need to do that, because the bgwriter is both the
> source of all the fsync requests and the guy who eventually executes
> them, so it can just update its local state directly.  If the two
> functions are separated, that gets a bit more complicated: the
> checkpoint process, even when not checkpointing, will need to absorb
> fsync requests every so often (maybe that could be driven off a latch,
> so we wake it up when the request queue is 25% full, or something like
> that?).  Or else the cleaning process will have to be in charge of
> issuing the fsyncs and the checkpoint process will have to request
> that it do so at the appropriate time.  Or maybe there's some third
> solution I'm not thinking of.  Anyway, it may not be a big problem in
> practice, just something I was worrying about...

Yes, they would still need to talk. But the good news is that they
only actually need to talk once per checkpoint cycle so we can buffer
them to a certain extent in shared memory to remove the worst part of
such contention.

Checkpointing needs a little more time in its diary to receive those
messages than it has right now, so there's no easy route.

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services