Re: "PANIC: could not open critical system index 2662" - twice

On Fri, May 5, 2023 at 7:50 PM Evgeny Morozov
<postgresql3(at)realityexists(dot)net> wrote:
> The OID of the bad DB ('test_behavior_638186279733138190') is 1414389 and I've uploaded base/1414389/pg_filenode.map and also base/5/2662 (in case that's helpful) as https://objective.realityexists.net/temp/pgstuff1.zip

Thanks. That pg_filenode.map looks healthy to me.

tmunro(at)build1:~/junk $ od -t x1 pg_filenode.map
0000000 17 27 59 00 11 00 00 00 eb 04 00 00 eb 04 00 00
0000020 e1 04 00 00 e1 04 00 00 e7 04 00 00 e7 04 00 00
0000040 df 04 00 00 df 04 00 00 14 0b 00 00 14 0b 00 00
0000060 15 0b 00 00 15 0b 00 00 4b 10 00 00 4b 10 00 00
0000100 4c 10 00 00 4c 10 00 00 82 0a 00 00 82 0a 00 00
0000120 83 0a 00 00 83 0a 00 00 8f 0a 00 00 8f 0a 00 00
0000140 90 0a 00 00 90 0a 00 00 62 0a 00 00 62 0a 00 00
0000160 63 0a 00 00 63 0a 00 00 66 0a 00 00 66 0a 00 00
...

hex(2662) is 0xa66, and we see 63 0a 00 00 followed by 63 0a 00 00 in
that last line as expected, so that rules out the idea that it's
somehow trashed that map file and points to the wrong relation file.

Next can you share the file base/1414389/2662? ("5" was from the
wrong database.)

> > Maybe you still have enough WAL if it happened recently?
>
> Maybe! What should I do with pg_waldump? I've never used it before.

Try something like:

pg_waldump -R 1663/1414389/2662 -F main 000000010000000000000001
000000010000000000000007

... but change that to the range of files you have in your pg_wal.