Re: "PANIC: could not open critical system index 2662" - twice

On Sun, May 7, 2023 at 10:18 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:

> On Mon, May 8, 2023 at 4:10 AM Evgeny Morozov
> <postgresql3(at)realityexists(dot)net> wrote:
> > On 6/05/2023 11:13 pm, Thomas Munro wrote:
> > > Would you like to try requesting FILE_COPY for a while and see if it
> eventually happens like that too?
> > Sure, we can try that.
>
> Maybe you could do some one way and some the other, so that we try to
> learn more?
>
> > Ubuntu 18.04.6
> > Kernel 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023
> > x86_64 x86_64 x86_64 GNU/Linux
> > zfsutils-linux package version 0.7.5-1ubuntu16.12 amd64
>
> I tried for a few hours to reproduce this by trying to make as many
> things as similar to yours as I could based on info in this thread
> (Linux: up-to-date Ubuntu 18.04 in vagrant which has nearly the same
> kernel 4.15.0-208-generic and a close but slightly different version
> of ancient ZFS 0.7.5-1ubuntu15, not sure why, ZFS: mirror (I used a
> pair of loopback files), recordsize=16kB, compression=lz4, PG:
> compiled from tag REL_15_2, data_checksums=on, full_page_writes=off,
> wal_recycle=off, wal_init_zero=off), with what I thought might be
> roughly what you're doing (creating three DBs, two clones of the
> first, with various modification at various points, with various
> overlapping activities, and then checking for catalog corruption). No
> cigar. Hrmph.
>

My first thought on this was along the lines of previous comments.
We are writing the initial pieces of the files for a new DATABASE.
Some flag is set, and then context is lost, and it ends up that a SAVE
happens
to an existing DATABASE.

So, my thought was a parallel set of UPDATE "type" commands that would
cause those types of pages for a stable/existing DB to be updated/written.

It could be as simple as creating temp tables in the other database (since
I believe pg_class was hit).

From a "rare" standpoint. That "feels" about right to me. It would pass
serial tests (like you are running).
Just a thought. Maybe a simple background script creating temp tables in
another DB.
Also, not sure if the OP has a set of things done after he creates the DB
that may help?

Kirk