Re: WAL segments removed from primary despite the fact that logical replication slot needs it.

From: Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com>
To: amit(dot)kapila16(at)gmail(dot)com
Cc: depesz(at)depesz(dot)com, pgsql-bugs(at)postgresql(dot)org
Subject: Re: WAL segments removed from primary despite the fact that logical replication slot needs it.
Date: 2022-10-19 02:00:32
Message-ID: 20221019.110032.2063205133677772338.horikyota.ntt@gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

At Tue, 18 Oct 2022 16:51:26 +0530, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote in
> On Mon, Oct 17, 2022 at 12:23 PM Kyotaro Horiguchi
> <horikyota(dot)ntt(at)gmail(dot)com> wrote:
> >
> > At Sun, 16 Oct 2022 10:35:17 +0530, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote in
> > > On Wed, Oct 5, 2022 at 8:54 PM hubert depesz lubaczewski
> > > > So, 4 files are missing.
> > > >
> > > > These were archived properly, and I tried to restore them from archive, and put
> > > > them in pg_wal, but even then pg12 was rejecting tries to connect to focal14
> > > > slot with the same message about "requested WAL segment
> > > > 000000010000CA0A00000049 has already been removed"
> > > >
> > >
> > > I think you are seeing this behavior because we update the
> > > lastRemovedSegNo before removing files in RemoveOldXlogFiles() and
> > > then we use that to give the error you are seeing.
> >
> > lastRemovedSegNo is updated once per one segment of removal. Four
> > files are lost in this case.
> >
>
> I didn't understand your response. I was saying the one possible
> reason why even after restoring files from the archive the error
> appears is because of the lastRemovedSegNo related check in function
> CheckXLogRemoved() and we update its value while removing old xlog
> files. From this behavior, it appears that somehow the server has only
> removed those files even though the reason is not clear yet.

I meant that if PostgreSQL did something wrong (that I don't
understand at all) there, the difference between lastRemovedSegNo and
the last actually removed segment won't be more than 1.

CheckXLogRevmoed warns for a segment logically already removed but was
physically able to be opend. On the other hand WalSndSegmentOpen
emits the same error earlier than CHeckXLogRemoved warns if the
segment is actually ENOENT regardless of the value of
lastRemovedSegNo.

One point that bothers me is it seems that the walsender was
killed. If the file is removed while the walsender is working, logical
replication worker receives the error and emits "ERROR: could not
receive data...has been removed" instead of suddenly disconnected as
in this case. Considering this together with the possibility that the
segments were removed by someone else, I suspencted virus scanners but
that has found to be false.. (I still don't know there's any virus
scanner that kills processes grabbing a suspectedly-malicious file.)

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

In response to

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Andy Fan 2022-10-19 02:21:42 Re: BUG #17650: For the sixth time, the clipping function in the 120 partition table planning stage fails
Previous Message Tom Lane 2022-10-18 21:07:41 Re: Aw: Re:  BUG #17647: 12.12 package has difference on ubuntu 18.04