Detecting File Damage & Inconsistencies

From: Simon Riggs <simon(at)2ndquadrant(dot)com>
To: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Detecting File Damage & Inconsistencies
Date: 2020-11-11 20:56:27
Message-ID: CANP8+jJSmLnsXG-q5EYGSjq1hC-RmSjCX9EnA2=AOhWc+uZVLA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

I would like to propose a few points that will help us detect file
damage, inconsistencies in files and track actions of users.

Optionally, we could add these fields onto the WAL commit record:
* 8-byte session start time (from MyStartTime)
* 2-byte pid (from MyProcPid)
* 4-byte user oid
Identified by a name flag bit, XACT_XINFO_HAS_USERINFO.
That allows us to match up transactions with the server log %c option.
Another option might allow text username to be added to each commit as well.

XLogLongPageHeaderData has 4 bytes of unused data because of
alignment. We could use that space for 1) the Xid Epoch value or 2) a
CRC value - since only WAL records are covered by a CRC, not page or
file headers. Perhaps we should add both?

There are also 2 bytes unused in the XLogRecord header (with 8 byte
alignment). We could optionally use that space for the pid that wrote
the record, but that's not compelling. What can we use those 2 bytes
for?

REINDEX VERIFY
After the new index is created, but before we drop the old index:
Check whether the two indexes match:
* checks whether the previous index had pointers to row versions that
don't exist
* checks whether the heap has rows that were not in the old index
This approach piggybacks on existing operations. AccessShareLock is
held on both indexes before the old one is dropped.

Other ideas are welcome.

Thoughts?

--
Simon Riggs http://www.EnterpriseDB.com/

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Victor Yegorov 2020-11-11 20:58:06 Re: Deleting older versions in unique indexes to avoid page splits
Previous Message John Naylor 2020-11-11 20:45:06 Re: cutting down the TODO list thread