Re: pg_receivewal starting position

Le jeudi 29 juillet 2021, 07:32:37 CEST Kyotaro Horiguchi a écrit :
> I didn't thought in details. But I forgot that ordinary SQL commands
> have been prohibited in physical replication connection. So we need a
> new replication command but it's not that a big deal.

Thank you for your feedback !

>
> > I don't see any reason not to make it work for logical replication
> > connections / slots, but it wouldn't be that useful since we can query
> > the database in that case.
>
> Ordinary SQL queries are usable on a logical replication slot so
> I'm not sure how logical replication connection uses the command.
> However, like you, I wouldn't bother restricting the command to
> physical replication, but perhaps the new command should return the
> slot type.
>

Ok done in the attached patch.

>
> I'm not sure it's worth adding complexity for such strictness.
> START_REPLICATION safely fails if someone steals the slot meanwhile.
> In the first place there's no means to protect a slot from others
> while idle. One possible problem is the case where START_REPLICATION
> successfully acquire the slot after the new command failed. But that
> case doesn't seem worse than the case someone advances the slot while
> absence. So I think READ_REPLICATION_SLOT is sufficient.
>

Ok, I implemented it like this. I tried to follow the pg_get_replication_slots
approach with regards to how to prevent concurrent modification while reading
the slot.

> > From pg_receivewal point of view, this would amount to:
> > - check if we currently have wal in the target directory.
> >
> > - if we do, proceed as currently done, by computing the start lsn and
> >
> > timeline from the last archived wal
> >
> > - if we don't, and we have a slot, run ACQUIRE_REPLICATION_SLOT. Use the
> >
> > restart_lsn as the start lsn if there is one, and don't provide a timeline
> >
> > - if we still don't have a start_lsn, fallback to using the current
> > server
> >
> > wal position as is done.
>
> That's pretty much it.

Great.

>
> > What do you think ? Which information should we provide about the slot ?
>
> We need the timeline id to start with when using restart_lsn. The
> current timeline can be used in most cases but there's a case where
> the LSN is historical.

Ok, see below.

>
> pg_receivewal doesn't send a replication status report when a segment
> is finished. So after pg_receivewal stops just after a segment is
> finished, the slot stays at the beginning of the last segment. Thus
> next time it will start from there, creating a duplicate segment.

I'm not sure I see where the problem is here. If we don't keep the segments in
pg_walreceiver target directory, then it would be the responsibility of
whoever moved them to make sure we don't have duplicates, or to handle them
gracefully.

Even if we were forcing a feedback after a segment is finished, there could
still be a problem if the feedback never made it to the server but the segment
was here. It might be interesting to send a feedback anyway.

Please find attached two patches implementing what we've been discussing.

Patch 0001 adds the new READ_REPLICATION_SLOT command.
It returns for a given slot the type, restart_lsn, flush_lsn,
restart_lsn_timeline and flush_lsn_timeline.
The timelines are determined by reading the current timeline history, and
finding the timeline where we may find the record. I didn't find explicit test
for eg IDENTIFY_SYSTEM so didn't write one either for this new command, but it
is tested indirectly in patch 0002.

Patch 0002 makes pg_receivewal use that command if we use a replication slot
and the command is available, and use the restart_lsn and restart_lsn_timeline
as a starting point. It also adds a small test to check that we start back
from the previous restart_lsn instead of the current flush position when our
destination directory does not contain any WAL file.

I also noticed we don't test following a timeline switch. It would probably be
good to add that, both for the case where we determine the previous timeline
from the archived segments and when it comes from the new command. What do you
think ?

Regards,

--
Ronan Dunklau