Re: BUG #16833: postgresql 13.1 process crash every hour

Thank you Tom for detailed instructions!
What I understood is that some specific query lead to database block
corruption which causes segfault.
I analyzed action log before segfault
1. database worked fine on pg13.0 for a few months
2. issue caused right after pg13.1 binaries upgrade
3. (this actually was wrong action) rollback pg13.0 binaries and tried to
start database on it and now got infinite segfault on startup
LOG: database system was interrupted while in recovery at 2021-01-22
14:27:49 UTC
HINT: This probably means that some data is corrupted and you will have to
use the last backup for recovery.
LOG: database system was not properly shut down; automatic recovery in
progress
LOG: redo starts at 49D/364DD528
LOG: startup process (PID 26) was terminated by signal 11: Segmentation
fault
LOG: aborting startup due to startup process failure
LOG: database system is shut down

Most disappointed fact that slave node also got corrupted wal block and
also unable to start.
So I have a chance to recover database with initdb+pgdump only.

Anyway I will try to compile pg13.1 binaries with --enable-debug and enable
all queries logging. Hope this will help with the investigation.
Thanks for your support!

пт, 22 янв. 2021 г. в 20:23, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:

> PG Bug reporting form <noreply(at)postgresql(dot)org> writes:
> > Process crash inside docker containter 2-3 times per hour without any
> > additional information
> > ./postgresql-Thu-00.log:2021-01-21 00:11:34 UTC [1]:
> user=,db=,app=,client=
> > LOG: server process (PID 20071) was terminated by signal 11:
> Segmentation
> > fault
>
> Hm, please see if you can get a stack trace:
>
>
> https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
>
> Also try to figure out what query(s) are causing the crash.
> (It's unlikely that the postmaster log doesn't provide more
> information than you've shared here.)
>
> > I can share postgresql.conf, process crash core dumps for analysis
>
> Core dumps are unlikely to help anyone else; they are too
> machine-specific. Not to mention that they might contain
> sensitive data. You'll need to examine them yourself.
>
> regards, tom lane
>