Re: POC: Cleaning up orphaned files using undo logs

Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:

> On Thu, Sep 9, 2021 at 8:33 PM Antonin Houska <ah(at)cybertec(dot)at> wrote:

> > The problem of the temporary undo log is that it's loaded into local buffers
> > and that backend can exit w/o flushing local buffers to disk, and thus we are
> > not guaranteed to find enough information when trying to discard the undo log
> > the backend wrote. I'm thinking about the following solutions:
> >
> > 1. Let the backend manage temporary undo log on its own (even the slot
> > metadata would stay outside the shared memory, and in particular the
> > insertion pointer could start from 1 for each session) and remove the
> > segment files at the same moment the temporary relations are removed.
> >
> > However, by moving the temporary undo slots away from the shared memory,
> > computation of oldestFullXidHavingUndo (see the PROC_HDR structure) would
> > be affected. It might seem that a transaction which only writes undo log
> > for temporary relations does not need to affect oldestFullXidHavingUndo,
> > but it needs to be analyzed thoroughly. Since oldestFullXidHavingUndo
> > prevents transactions to be truncated from the CLOG too early, I wonder if
> > the following is possible (This scenario is only applicable to the zheap
> > storage engine [1], which is not included in this patch, but should already
> > be considered.):
> >
> > A transaction creates a temporary table, does some (many) changes and then
> > gets rolled back. The undo records are being applied and it takes some
> > time. Since XID of the transaction did not affect oldestFullXidHavingUndo,
> > the XID can disappear from the CLOG due to truncation.
> >
>
> By above do you mean to say that in zheap code, we don't consider XIDs
> that operate on temp table/undo for oldestFullXidHavingUndo?

I was referring to the code

/* We can't process temporary undo logs. */
if (log->meta.persistence == UNDO_TEMP)
continue;

in undodiscard.c:UndoDiscard().

>
> > However zundo.c in
> > [1] indicates that the transaction status *is* checked during undo
> > execution, so we might have a problem.
> >
>
> It would be easier to follow if you can tell which exact code are you
> referring here?

In meant the call of TransactionIdDidCommit() in
zundo.c:zheap_exec_pending_rollback().

--
Antonin Houska
Web: https://www.cybertec-postgresql.com