Re: checkpointer continuous flushing

On Sat, Jan 9, 2016 at 7:10 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:
>
> On 2016-01-09 19:05:54 +0530, Amit Kapila wrote:
> > Right that won't be acceptable, however I think with your latest
> > proposal [1]
>
> Sure, that'd address that problem.
>
>
> > [...] think that idea will help to mitigate the problem of backend and
> > bgwriter writes as well. In that, can't we do it with the help of
> > existing infrastructure of *pendingOpsTable* and
> > *CheckpointerShmem->requests[]*, as already the flush requests are
> > remembered in those structures, we can use those to apply your idea to
> > issue flush requests.
>
> Hm, that might be possible. But that might have some bigger implications
> - we currently can issue thousands of flush requests a second, without
> much chance of merging. I'm not sure it's a good idea to overlay that
> into the lower frequency pendingOpsTable.
>

In that case, we can have unified structure to remember flush requests
rather than backend and bgwriter noting that information in
CheckpointerShmem and checkpointer in pendingOpsTable. I understand
there are some benefits of having pendingOpsTable, but having a
common structure seems to be more beneficial and in particular
because it can be used for the purpose of flush hints.

Now, I am sure we can invent a new way of tracking the flush
requests for flush hints, but I think we might want to consider why
can't we have one unified way of tracking the flush requests which
can be used both for *flush* and *flush hints*.

> Backends having to issue
> fsyncs because the pending fsync queue is full is darn expensive. In
> contrast to that a 'flush hint' request getting lost doesn't cost that
> much.
>

In general, I think the cases where backends have to do flush should
be less as the size of fsync queue is NBuffers and we take care of
handling duplicate fsync requests for the same buffer.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com