Re: Interesting fail when migrating Pg from Ubuntu Bionic to Focal

On Mon, Mar 07, 2022 at 09:54:22AM -0800, Andres Freund wrote:
> > Initially select didn't break anything, but when I tuned down
> > jit_above_cost so that it will kick in - got fails immediately.
> Could you set jit_debugging_support=on and show a backtrace with that?

Here you go:
Program received signal SIGSEGV, Segmentation fault.
0x0000fffe7587b6c8 in deform_0_8 ()
(gdb) bt
#0 0x0000fffe7587b6c8 in deform_0_8 ()
#1 0x0000fffe7587b5f4 in evalexpr_0_7 ()
#2 0x0000aaaab67e6f7c in ExecEvalExprSwitchContext (isNull=0xfffff7956217, econtext=0xaaaaf6014608, state=0xaaaaf6014ab0) at ./build/../src/include/executor/executor.h:316
#3 ExecProject (projInfo=0xaaaaf6014aa8) at ./build/../src/include/executor/executor.h:350
#4 ExecScan (node=<optimized out>, accessMtd=0xaaaab68051b8 <SeqNext>, recheckMtd=0xaaaab6805248 <SeqRecheck>) at ./build/../src/backend/executor/execScan.c:239
#5 0x0000aaaab67ded5c in ExecProcNode (node=0xaaaaf60143b8) at ./build/../src/include/executor/executor.h:242
#6 ExecutePlan (execute_once=<optimized out>, dest=0xaaaaf5fc15e8, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0xaaaaf60143b8, estate=0xaaaaf5e01360)
at ./build/../src/backend/executor/execMain.c:1632
#7 standard_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=<optimized out>, count=0, execute_once=<optimized out>) at ./build/../src/backend/executor/execMain.c:350
#8 0x0000ffffa87751fc in pgss_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=ForwardScanDirection, count=0, execute_once=true) at ./build/../contrib/pg_stat_statements/pg_stat_statements.c:893
#9 0x0000ffffa875f688 in explain_ExecutorRun (queryDesc=0xaaaaf5ef4130, direction=ForwardScanDirection, count=0, execute_once=true) at ./build/../contrib/auto_explain/auto_explain.c:320
#10 0x0000aaaab69354c8 in PortalRunSelect (portal=portal(at)entry=0xaaaaf5e83ef0, forward=forward(at)entry=true, count=0, count(at)entry=9223372036854775807, dest=dest(at)entry=0xaaaaf5fc15e8) at ./build/../src/backend/tcop/pquery.c:938
#11 0x0000aaaab6936a64 in PortalRun (portal=portal(at)entry=0xaaaaf5e83ef0, count=count(at)entry=9223372036854775807, isTopLevel=isTopLevel(at)entry=true, run_once=run_once(at)entry=true, dest=dest(at)entry=0xaaaaf5fc15e8, altdest=altdest(at)entry=0xaaaaf5fc15e8,
completionTag=completionTag(at)entry=0xfffff7956848 "") at ./build/../src/backend/tcop/pquery.c:779
#12 0x0000aaaab6932a18 in exec_simple_query (query_string=query_string(at)entry=0xaaaaf5d9af90 "select p.proname, (SELECT rolname from pg_catalog.pg_roles where oid = p.proowner) from pg_proc p;") at ./build/../src/backend/tcop/postgres.c:1215
#13 0x0000aaaab69336ac in PostgresMain (argc=<optimized out>, argv=argv(at)entry=0xaaaaf5df6f18, dbname=<optimized out>, username=<optimized out>) at ./build/../src/backend/tcop/postgres.c:4271
#14 0x0000aaaab68ba768 in BackendRun (port=0xaaaaf5df0910, port=0xaaaaf5df0910) at ./build/../src/backend/postmaster/postmaster.c:4510
#15 BackendStartup (port=0xaaaaf5df0910) at ./build/../src/backend/postmaster/postmaster.c:4193
#16 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1725
#17 0x0000aaaab68bb74c in PostmasterMain (argc=<optimized out>, argv=<optimized out>) at ./build/../src/backend/postmaster/postmaster.c:1398
#18 0x0000aaaab664c8a8 in main (argc=5, argv=0xaaaaf5d94ea0) at ./build/../src/backend/main/main.c:228

> > > FWIW, I went to the trouble of installing Ubuntu Focal on my
> > > raspberry pi to see if I could duplicate this, and I couldn't.
> > > However, what you get from a fresh install now is
> > > $ dpkg -l | grep libllvm
> > > ii libllvm10:arm64 1:10.0.0-4ubuntu1 arm64 Modular compiler and toolchain technologies, runtime library
> > > not 9.0.1. I also found that Fedora 31/aarch64 is still downloadable, and
> > > that does contain LLVM 9 ... and I could not reproduce it there either.
> >
> > Soo... plot thickens.
> >
> > Looks that pg 12 supplied by pgdg required libllvm9:
> What are the libllvm package versions on which you can reproduce the crash?

Hmm .. not sure I fully understand. Here are all packages that I think are relevant:

root(at)c999-pg1302:~# dpkg -l | grep -iE 'postgresql-12|llvm|clang'
ii libllvm9:arm64 1:9.0.1-12 arm64 Modular compiler and toolchain technologies, runtime library
ii postgresql-12 12.9-2.pgdg20.04+1 arm64 The World's Most Advanced Open Source Relational Database
ii postgresql-12-dbgsym 12.9-2.pgdg20.04+1 arm64 debug symbols for postgresql-12
ii postgresql-12-pg-collkey 0.5.1-1insops1 arm64 ICU collation function wrapper for PostgreSQL 12
ii postgresql-12-postgis-2.5 2.5.5+dfsg-1.pgdg20.04+2 arm64 Geographic objects support for PostgreSQL 12
ii postgresql-12-postgis-2.5-scripts 2.5.5+dfsg-1.pgdg20.04+2 all Geographic objects support for PostgreSQL 12 -- SQL scripts
ii postgresql-12-repack 1.4.7-2.pgdg20.04+1 arm64 reorganize tables in PostgreSQL databases with minimal locks
ii postgresql-12-system-stats 1.0.0-1 arm64 A packaged version of EnterpriseDB's system_stats postgres extension

Hope it helps.

Best regards,

depesz