Re: POC: Cleaning up orphaned files using undo logs

On Thu, Sep 9, 2021 at 8:33 PM Antonin Houska <ah(at)cybertec(dot)at> wrote:
>
> The cfbot complained that the patch series no longer applies, so I've rebased
> it and also tried to make sure that the other flags become green.
>
> One particular problem was that pg_upgrade complained that "live undo data"
> remains in the old cluster. I found out that the temporary undo log causes the
> problem, so I've adjusted the query in check_for_undo_data() accordingly until
> the problem gets fixed properly.
>
> The problem of the temporary undo log is that it's loaded into local buffers
> and that backend can exit w/o flushing local buffers to disk, and thus we are
> not guaranteed to find enough information when trying to discard the undo log
> the backend wrote. I'm thinking about the following solutions:
>
> 1. Let the backend manage temporary undo log on its own (even the slot
> metadata would stay outside the shared memory, and in particular the
> insertion pointer could start from 1 for each session) and remove the
> segment files at the same moment the temporary relations are removed.
>
> However, by moving the temporary undo slots away from the shared memory,
> computation of oldestFullXidHavingUndo (see the PROC_HDR structure) would
> be affected. It might seem that a transaction which only writes undo log
> for temporary relations does not need to affect oldestFullXidHavingUndo,
> but it needs to be analyzed thoroughly. Since oldestFullXidHavingUndo
> prevents transactions to be truncated from the CLOG too early, I wonder if
> the following is possible (This scenario is only applicable to the zheap
> storage engine [1], which is not included in this patch, but should already
> be considered.):
>
> A transaction creates a temporary table, does some (many) changes and then
> gets rolled back. The undo records are being applied and it takes some
> time. Since XID of the transaction did not affect oldestFullXidHavingUndo,
> the XID can disappear from the CLOG due to truncation.
>

By above do you mean to say that in zheap code, we don't consider XIDs
that operate on temp table/undo for oldestFullXidHavingUndo?

> However zundo.c in
> [1] indicates that the transaction status *is* checked during undo
> execution, so we might have a problem.
>

It would be easier to follow if you can tell which exact code are you
referring here?

--
With Regards,
Amit Kapila.