Re: [Incident report]Backend process crashed when executing 2pc transaction

Hello,

On Wed, Nov 27, 2019 at 8:59 PM LIANGBO <liangboa(at)suning(dot)com> wrote:
> I've met the following problem in our product environment. We tried to reproduce the problem, but because of the low probability of occurrence, we could not reproduce it.
> 1. phenomenon
> Backend process crashed when executing 2pc transaction in citus.
>
> 2. Occurrence condition
> Distributed transaction in business SQL
>
> PostgreSQL：10.7
> citus：7.4.1
> OS：RHEL6.3
>
> 3.2 stacktrace
>
> (gdb) bt
> #0 0x000000369e232625 in raise () from /lib64/libc.so.6
> #1 0x000000369e233e05 in abort () from /lib64/libc.so.6
> #2 0x000000369e270537 in __libc_message () from /lib64/libc.so.6
> #3 0x000000369e275f4e in malloc_printerr () from /lib64/libc.so.6
> #4 0x000000369e278cf0 in _int_free () from /lib64/libc.so.6
> #5 0x00000000004ff947 in XLogReaderFree (state=0x1a403a8) at xlogreader.c:141
> #6 0x00000000004e4387 in XlogReadTwoPhaseData (lsn=32886947137584, buf=0x7fff4a40ec38, len=0x0) at twophase.c:1341
> #7 0x00000000004e5849 in FinishPreparedTransaction (gid=0x19d7830 "citus_0_35791_4207001212_1287199", isCommit=1 '\001') at twophase.c:1411
> #8 0x000000000072d601 in standard_ProcessUtility (pstmt=0x19d7ba0, queryString=0x19d6e48 "COMMIT PREPARED 'citus_0_35791_4207001212_1287199'", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
> queryEnv=0x0, dest=0x19d7c80, completionTag=0x7fff4a40f260 "") at utility.c:460
> #9 0x00007f63a1ae97e1 in multi_ProcessUtility (pstmt=0x19d7ba0, queryString=0x19d6e48 "COMMIT PREPARED 'citus_0_35791_4207001212_1287199'", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
> queryEnv=0x0, dest=0x19d7c80, completionTag=0x7fff4a40f260 "") at executor/multi_utility.c:254
> #10 0x00007f63a11e1178 in pgss_ProcessUtility (pstmt=0x19d7ba0, queryString=0x19d6e48 "COMMIT PREPARED 'citus_0_35791_4207001212_1287199'", context=PROCESS_UTILITY_TOPLEVEL, params=0x0,

Have you considered *also* reporting this to Citus developers, because
while the crash seems to have occurred in the core PostgreSQL code
they may have a better chance reproducing this if at all.

Thanks,
Amit