Re: POC: Cleaning up orphaned files using undo logs

On Mon, Sep 20, 2021 at 10:55 AM Antonin Houska <ah(at)cybertec(dot)at> wrote:
>
> Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> > On Thu, Sep 9, 2021 at 8:33 PM Antonin Houska <ah(at)cybertec(dot)at> wrote:
>
> > > The problem of the temporary undo log is that it's loaded into local buffers
> > > and that backend can exit w/o flushing local buffers to disk, and thus we are
> > > not guaranteed to find enough information when trying to discard the undo log
> > > the backend wrote. I'm thinking about the following solutions:
> > >
> > > 1. Let the backend manage temporary undo log on its own (even the slot
> > > metadata would stay outside the shared memory, and in particular the
> > > insertion pointer could start from 1 for each session) and remove the
> > > segment files at the same moment the temporary relations are removed.
> > >
> > > However, by moving the temporary undo slots away from the shared memory,
> > > computation of oldestFullXidHavingUndo (see the PROC_HDR structure) would
> > > be affected. It might seem that a transaction which only writes undo log
> > > for temporary relations does not need to affect oldestFullXidHavingUndo,
> > > but it needs to be analyzed thoroughly. Since oldestFullXidHavingUndo
> > > prevents transactions to be truncated from the CLOG too early, I wonder if
> > > the following is possible (This scenario is only applicable to the zheap
> > > storage engine [1], which is not included in this patch, but should already
> > > be considered.):
> > >
> > > A transaction creates a temporary table, does some (many) changes and then
> > > gets rolled back. The undo records are being applied and it takes some
> > > time. Since XID of the transaction did not affect oldestFullXidHavingUndo,
> > > the XID can disappear from the CLOG due to truncation.
> > >
> >
> > By above do you mean to say that in zheap code, we don't consider XIDs
> > that operate on temp table/undo for oldestFullXidHavingUndo?
>
> I was referring to the code
>
> /* We can't process temporary undo logs. */
> if (log->meta.persistence == UNDO_TEMP)
> continue;
>
> in undodiscard.c:UndoDiscard().
>

Here, I think it will just skip undo of temporary undo logs and
oldestFullXidHavingUndo should be advanced after skipping it.

> >
> > > However zundo.c in
> > > [1] indicates that the transaction status *is* checked during undo
> > > execution, so we might have a problem.
> > >
> >
> > It would be easier to follow if you can tell which exact code are you
> > referring here?
>
> In meant the call of TransactionIdDidCommit() in
> zundo.c:zheap_exec_pending_rollback().
>

IIRC, this should be called for temp tables after they have exited as
this is only to apply the pending undo actions if any, and in case of
temporary undo after session exit, we shouldn't need it.

I am not able to understand what exact problem you are facing for temp
tables after the session exit. Can you please explain it a bit more?

--
With Regards,
Amit Kapila.