Re: "PANIC: could not open critical system index 2662" - twice

On Sat, May 6, 2023 at 9:58 PM Evgeny Morozov
<postgresql3(at)realityexists(dot)net> wrote:
> Right - I should have realised that! base/1414389/2662 is indeed all
> nulls, 32KB of them. I included the file anyway in
> https://objective.realityexists.net/temp/pgstuff2.zip

OK so it's not just page 0, you have 32KB or 4 pages of all zeroes.
That's the expected length of that relation when copied from the
initial template, and consistent with the pg_waldump output (it uses
FPIs to copy blocks 0-3). We can't see the block contents but we know
that block 2 definitely is not all zeroes at that point because there
are various modifications to it, which not only write non-zeroes but
must surely have required a sane page 0.

So it does indeed look like something unknown has replaced 32KB of
data with 32KB of zeroes underneath us. Are there more non-empty
files that are all-zeroes? Something like this might find them:

for F in base/1414389/*
do
if [ -s $F ] && ! xxd -p $F | grep -qEv '^(00)*$' > /dev/null
then
echo $F
fi
done