Re: POC: Cleaning up orphaned files using undo logs

On Mon, Jul 1, 2019 at 1:24 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
>
>
> 1. Renamed UndoPersistence to UndoLogCategory everywhere, and add a
> fourth category UNDO_SHARED where transactions can write 'out of band'
> data that relates to more than one transaction.
>
> 2. Introduced a new RMGR callback rm_undo_status. It is used to
> decide when record sets in the UNDO_SHARED category should be
> discarded (instead of the usual single xid-based rules). The possible
> answers are "discard me now!", "ask me again when a given XID is all
> visible", and "ask me again when a given XID is no longer running".
>
> 3. Recognise UNDO_SHARED record set boundaries differently. Whereas
> undolog.c recognises transaction boundaries automatically for the
> other categories (UNDO_PERMANENT, UNDO_UNLOGGED, UNDO_TEMP), for
> UNDO_SHARED the
>
> 4. Add some quick-and-dirty throw-away test stuff to demonstrate
> that. SELECT test_multixact([1234, 2345]) will create a new record
> set that will survive until the given array of transactions is no
> longer running, and then it'll be discarded. You can see that with
> SELECT * FROM undoinspect('shared'). Or look at SELECT
> pg_stat_undo_logs. This test simply writes all the xids into its
> payload, and then has an rm_undo_status function that returns the
> first xid it finds in the list that is still running, or if none are
> running returns UNDO_STATUS_DISCARD.
>
> Currently you can only return UNDO_STATUS_WAIT_XMIN so wait for an xid
> to be older than the oldest xmin; presumably it'd be useful to be able
> to discard as soon as an xid is no longer active, which could be a bit
> sooner.
>
> Another small change: several people commented that
> UndoLogIsDiscarded(ptr) ought to have some kind of fast path that
> doesn't acquire locks since it'll surely be hammered. Here's an
> attempt at that that provides an inlined function that uses a
> per-backend recent_discard to avoid doing more work in the (hopefully)
> common case that you mostly encounter discarded undo pointers. I hope
> this change will show up in profilers in some zheap workloads but this
> hasn't been tested yet.
>
> Another small change/review: the function UndoLogGetNextInsertPtr()
> previously took a transaction ID, but I'm not sure if that made sense,
> I need to think about it some more.
>
> I pulled the latest patches pulled in from the "undoprocessing" branch
> as of late last week, and most of the above is implemented as fixup
> commits on top of that.
>
> Next I'm working on DBA facilities for forcing undo records to be
> discarded (which consists mostly of sorting out the interlocking to
> make that work safely). And also testing facilities for simulating
> undo log switching (when you fill up each log and move to another one,
> which are rare code paths run, so we need a good way to make them not
> rare).
>

In 0003-Add-undo-log-manager

/* If we discarded everything, the slot can be given up. */
+ if (entirely_discarded)
+ free_undo_log_slot(slot);

I have noticed that when the undo log was detached and it was full
then if we discard complete log we release its slot. But, what is
bothering me is should we add that log to the free list? Or I am
missing something?

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com