Re: WALWriter active during recovery

Hi,

On 2014-12-15 18:51:44 +0000, Simon Riggs wrote:
> Currently, WALReceiver writes and fsyncs data it receives. Clearly,
> while we are waiting for an fsync we aren't doing any other useful
> work.

Well, it can still buffer data on the network level, but there's
definitely limits to that. So I can see this as being useful.

> Following patch starts WALWriter during recovery and makes it
> responsible for fsyncing data, allowing WALReceiver to progress other
> useful actions.
>
> At present this is a WIP patch, for code comments only. Don't bother
> with anything other than code questions at this stage.
>
> Implementation questions are
>
> * How should we wake WALReceiver, since it waits on a poll(). Should
> we use SIGUSR1, which is already used for latch waits, or another
> signal?

It's not entirely trivial, but also not hard, to make it use the latch
code for waiting. It'd probably end up requiring less code because then
we could just scratch libqpwalreceiver.c:libpq_select().

> * Should we introduce some pacing delays if the WALreceiver gets too
> far ahead of apply?

Hm. Why don't we simply start fsyncing in the receiver itself at regular
intervals? If already synced that's cheap, if not, it'll pace us.

Greetings,

Andres Freund