Assertion failure on PG15 with modified test_shm_mq test

Hello,

I've a slightly modified version of test_shm_mq, that I changed to include
a shared fileset. The motivation to do that came because I hit an
assertion failure with PG15 while doing some development work on BDR and I
suspected it to be a PG15 bug.

The stack trace looks as below:

(lldb) bt
* thread #1
* frame #0: 0x00007ff8187b100e libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x00007ff8187e71ff libsystem_pthread.dylib`pthread_kill + 263
frame #2: 0x00007ff818732d24 libsystem_c.dylib`abort + 123
frame #3: 0x000000010fce1bab
postgres`ExceptionalCondition(conditionName="pgstat_is_initialized &&
!pgstat_is_shutdown", errorType="FailedAssertion", fileName="pgstat.c",
lineNumber=1227) at assert.c:69:2
frame #4: 0x000000010fb06412 postgres`pgstat_assert_is_up at
pgstat.c:1227:2
frame #5: 0x000000010fb0d2c7
postgres`pgstat_get_entry_ref(kind=PGSTAT_KIND_DATABASE, dboid=0, objoid=0,
create=true, created_entry=0x0000000000000000) at pgstat_shmem.c:406:2
frame #6: 0x000000010fb07579
postgres`pgstat_prep_pending_entry(kind=PGSTAT_KIND_DATABASE, dboid=0,
objoid=0, created_entry=0x0000000000000000) at pgstat.c:1068:14
frame #7: 0x000000010fb09cce
postgres`pgstat_prep_database_pending(dboid=0) at pgstat_database.c:327:14
frame #8: 0x000000010fb09dff
postgres`pgstat_report_tempfile(filesize=0) at pgstat_database.c:179:10
frame #9: 0x000000010fa8dbe9
postgres`ReportTemporaryFileUsage(path="base/pgsql_tmp/pgsql_tmp17312.0.fileset/test_mq_sharefile.0",
size=0) at fd.c:1521:2
frame #10: 0x000000010fa8db3c
postgres`PathNameDeleteTemporaryFile(path="base/pgsql_tmp/pgsql_tmp17312.0.fileset/test_mq_sharefile.0",
error_on_failure=false) at fd.c:1945:3
frame #11: 0x000000010fa8d3a8
postgres`unlink_if_exists_fname(fname="base/pgsql_tmp/pgsql_tmp17312.0.fileset/test_mq_sharefile.0",
isdir=false, elevel=15) at fd.c:3674:3
frame #12: 0x000000010fa8d270
postgres`walkdir(path="base/pgsql_tmp/pgsql_tmp17312.0.fileset",
action=(postgres`unlink_if_exists_fname at fd.c:3663),
process_symlinks=false, elevel=15) at fd.c:3573:5
frame #13: 0x000000010fa8d0e2
postgres`PathNameDeleteTemporaryDir(dirname="base/pgsql_tmp/pgsql_tmp17312.0.fileset")
at fd.c:1689:2
frame #14: 0x000000010fa91ac1
postgres`FileSetDeleteAll(fileset=0x0000000119240870) at fileset.c:165:3
frame #15: 0x000000010fa92b08
postgres`SharedFileSetOnDetach(segment=0x00007f93ff00a7c0,
datum=4716759152) at sharedfileset.c:119:3
frame #16: 0x000000010fa96b00
postgres`dsm_detach(seg=0x00007f93ff00a7c0) at dsm.c:801:3
frame #17: 0x000000010fa96f51 postgres`dsm_backend_shutdown at
dsm.c:738:3
frame #18: 0x000000010fa99402 postgres`shmem_exit(code=1) at ipc.c:259:2
frame #19: 0x000000010fa99227 postgres`proc_exit_prepare(code=1) at
ipc.c:194:2
frame #20: 0x000000010fa99133 postgres`proc_exit(code=1) at ipc.c:107:2
frame #21: 0x000000010fce318c postgres`errfinish(filename="postgres.c",
lineno=3204, funcname="ProcessInterrupts") at elog.c:661:3
frame #22: 0x000000010fad7c51 postgres`ProcessInterrupts at
postgres.c:3201:4
frame #23: 0x000000011924d85b
test_shm_mq.so`test_shm_mq_main(main_arg=1155036180) at worker.c:159:2
frame #24: 0x000000010f9da11e postgres`StartBackgroundWorker at
bgworker.c:858:2
frame #25: 0x000000010f9e80b4
postgres`do_start_bgworker(rw=0x00007f93ef904080) at postmaster.c:5823:4
frame #26: 0x000000010f9e2524 postgres`maybe_start_bgworkers at
postmaster.c:6047:9
frame #27: 0x000000010f9e0e63
postgres`sigusr1_handler(postgres_signal_arg=30) at postmaster.c:5204:3
frame #28: 0x00007ff8187fcdfd libsystem_platform.dylib`_sigtramp + 29
frame #29: 0x00007ff8187b2d5b libsystem_kernel.dylib`__select + 11
frame #30: 0x000000010f9e268d postgres`ServerLoop at
postmaster.c:1770:13
frame #31: 0x000000010f9e0157 postgres`PostmasterMain(argc=8,
argv=0x0000600002f30190) at postmaster.c:1478:11
frame #32: 0x000000010f8bc930 postgres`main(argc=8,
argv=0x0000600002f30190) at main.c:202:3
frame #33: 0x000000011f7d651e dyld`start + 462

I notice that pgstat_shutdown_hook() is registered as a before-shmem-exit
callback. The callback is responsible for detaching from the pgstat shared
memory segment. But looks like other parts of the system still expect it to
be available during later stages of proc exit.

It's not clear to me if pgstat shutdown should happen later or code that
gets executed later in the cycle should not try to use pgstat. It's also
entirely possible that my usage of SharedFileSet is completely wrong. If
that's the case, please let me know the mistake in the usage.

Patch modifying the test case is attached. In order to reproduce the
problem quickly, I added a CHECK_FOR_INTERRUPTS() in the test, but I don't
see why that would be a bad coding practice.

Thanks,
Pavan