Re: CDC/ETL system on top of logical replication with pgoutput, custom client

On Wed, Oct 25, 2023 at 4:23 PM José Neves <rafaneves3(at)msn(dot)com> wrote:
>
> Ok, I see. In that situation is safe indeed, as the offset is lower than the current transaction commit.
> But I think that I asked the wrong question. I guess that the right question is: Can we receive a keep-alive message with an LSN offset bigger than the commit of the open or following transactions?
> Something like:
>
> BEGIN LSN001
> INSERT LSN002
> KEEP LIVE LSN006
> UPDATE LSN004
> COMMIT LSN005
>
> Or:
>
> KEEP LIVE LSN006
> BEGIN LSN001
> INSERT LSN002
> UPDATE LSN004
> COMMIT LSN005
> KEEP LIVE LSN007
>

AFAIU the code in walsender this isn't possible. Keep alive sends the
LSN of the last WAL record it read (sentPtr). Upon reading a commit
WAL record, the whole transaction is decoded. Till that point sentPtr
is not updated.

Please take a look at XLogSendLogical(void) and the places where
WalSndKeepaliveIfNecessary() is called.

--
Best Wishes,
Ashutosh Bapat