Re: pg_basebackup may fail to send feedbacks.

On Fri, Feb 6, 2015 at 3:22 PM, Kyotaro HORIGUCHI
<horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote:
> Sorry, I misunderstood that.
>
>> > At Wed, 4 Feb 2015 19:22:39 +0900, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote in <CAHGQGwGudGCMnHZinkd37i+JijDkruEcrea1NCRs1MMtE3rOFQ(at)mail(dot)gmail(dot)com>
>> >> On Wed, Feb 4, 2015 at 4:58 PM, Kyotaro HORIGUCHI
>> >> <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote:
>> >> > I'm very sorry for confused report. The problem found in 9.4.0
>> >> > and the diagnosis was mistakenly done on master.
>> >> >
>> >> > 9.4.0 has no problem of feedback delay caused by slow xlog
>> >> > receiving on pg_basebackup mentioned in the previous mail. But
>> >> > the current master still has this problem.
>> >>
>> >> Seems walreceiver has the same problem. No?
>> >
>> > pg_receivexlog.c would have the same problem since it uses the
>> > same function with pg_basebackup.c.
>> >
>> > The correspondent of HandleCopyStream in wansender is
>> > WalReceiverMain, and it doesn't seem to have the same kind of
>> > loop shown below. It seems to surely send feedback per one
>> > record.
>> >
>> > | r = stream_reader();
>> > | while (r > 0)
>> > | {
>> > | ... wal record processing stuff without sending feedback..
>> > | r = stream_reader();
>> > | }
>>
>> WalReceiverMain() has the similar code as follows.
>>
>> len = walrcv_receive(NAPTIME_PER_CYCLE, &buf);
>> if (len != 0)
>> {
>> for (;;)
>> {
>> if (len > 0)
>> {
>> ....
>> len = walrcv_receive(0, &buf);
>> }
>> }
>
> The loop seems a bit different but eventually the same about this
> discussion.
>
> 408> len = walrcv_receive(NAPTIME_PER_CYCLE, &buf);
> 409> if (len != 0)
> 410> {
> 415> for (;;)
> 416> {
> 417> if (len > 0)
> 418> {
> 425> XLogWalRcvProcessMsg(buf[0], &buf[1], len - 1);
> 426> }
> 427-438> else {break;}
> 439> len = walrcv_receive(0, &buf);
> 440> }
> 441> }
>
> XLogWalRcvProcessMsg doesn't send feedback when processing
> 'w'=WAL record packet. So the same thing as pg_basebackup and
> pg_receivexlog will occur on walsender.
>
> Exiting the for(;;) loop by TimestampDifferenceExceeds just
> before line 439 is an equivalent measure to I poposed for
> receivelog.c, but calling XLogWalRcvSendHSFeedback(false) there
> is seemingly simpler (but I feel a bit uncomfortable for the
> latter)

I'm concerned about the performance degradation by calling
getimeofday() per a record.

> Or other measures?

Introduce the maximum number of records that we can receive and
process between feedbacks? For example, we can change
XLogWalRcvSendHSFeedback() so that it's called per at least
8 records. I'm not sure what number is good, though...

Regards,

--
Fujii Masao