Re: BUG #13490: Segmentation fault on pg_stat_activity

Hello Michael,
Here is the offending query and gdb session/stacktrace output. Please
let me know if we can provide anything else from gdb or logs that can be
anonymized.

*Query:*
2015-07-11 12:57:41 UTC [12803-7] LOG: server process (PID 20696) was
terminated by signal 11: Segmentation fault
2015-07-11 12:57:41 UTC [12803-8] DETAIL: Failed process was running:
SELECT COUNT(*) FROM pg_stat_activity WHERE pid <> pg_backend_pid()

*GDB:*
root(at)hostname:/var/lib/postgresql/9.5/main# gdb -q -c
/var/lib/postgresql/9.5/main/core /usr/lib/postgresql/9.5/bin/postgres
Reading symbols from /usr/lib/postgresql/9.5/bin/postgres...Reading symbols
from
/usr/lib/debug/.build-id/7b/273fca376bfad4ea4bb0c4d3099d62ecc9c124.debug...done.
done.
[New LWP 16028]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: postgres databasename 127.0.0.1(42063)
BIND '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 get_tle_by_resno (tlist=0x7fd0d5da27c0, resno=resno(at)entry=6) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/parser/parse_relation.c:2832
2832
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/parser/parse_relation.c:
No such file or directory.
(gdb) bt
#0 get_tle_by_resno (tlist=0x7fd0d5da27c0, resno=resno(at)entry=6) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/parser/parse_relation.c:2832
#1 0x00007fd0d47cb9dd in pullup_replace_vars_callback (var=0x7fd0d5d9e958,
context=0x7fff52170620) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/optimizer/prep/prepjointree.c:2074
#2 0x00007fd0d481c3ea in replace_rte_variables_mutator (node=<optimized
out>, context=0x7fff52170620) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/rewrite/rewriteManip.c:1149
#3 0x00007fd0d478152c in expression_tree_mutator (node=0x7fd0d5d9e908,
mutator=0x7fd0d481c390 <replace_rte_variables_mutator>,
context=0x7fff52170620) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/nodes/nodeFuncs.c:2769
#4 0x00007fd0d47812b3 in expression_tree_mutator (node=<optimized out>,
mutator=0x7fd0d481c390 <replace_rte_variables_mutator>,
context=0x7fff52170620) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/nodes/nodeFuncs.c:2675
#5 0x00007fd0d481cc64 in replace_rte_variables (node=<optimized out>,
target_varno=<optimized out>, sublevels_up=sublevels_up(at)entry=0,
callback=callback(at)entry=0x7fd0d47cb880 <pullup_replace_vars_callback>,
callback_arg=callback_arg(at)entry=0x7fff521706c0,
outer_hasSubLinks=0x7fd0d5d30d6e "")
at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/rewrite/rewriteManip.c:1115
#6 0x00007fd0d47cd1c7 in pullup_replace_vars (context=0x7fff521706c0,
expr=<optimized out>) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/optimizer/prep/prepjointree.c:1982
#7 pull_up_simple_subquery (deletion_ok=<optimized out>,
containing_appendrel=0x0, lowest_nulling_outer_join=0x0,
lowest_outer_join=0x0, rte=0x7fd0d5d30ea8, jtnode=<optimized out>,
root=0x7fd0d5d9ee48) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/optimizer/prep/prepjointree.c:1030
#8 pull_up_subqueries_recurse (root=root(at)entry=0x7fd0d5d9ee48,
jtnode=<optimized out>, lowest_outer_join=lowest_outer_join(at)entry=0x0,
lowest_nulling_outer_join=lowest_nulling_outer_join(at)entry=0x0,
containing_appendrel=containing_appendrel(at)entry=0x0, deletion_ok=<optimized
out>)
at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/optimizer/prep/prepjointree.c:696
#9 0x00007fd0d47cc989 in pull_up_subqueries_recurse
(root=root(at)entry=0x7fd0d5d9ee48,
jtnode=0x7fd0d5d9e6c0, lowest_outer_join=lowest_outer_join(at)entry=0x0,
lowest_nulling_outer_join=lowest_nulling_outer_join(at)entry=0x0,
containing_appendrel=containing_appendrel(at)entry=0x0, deletion_ok=<optimized
out>,
deletion_ok(at)entry=0 '\000') at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/optimizer/prep/prepjointree.c:762
#10 0x00007fd0d47cd639 in pull_up_subqueries (root=root(at)entry=0x7fd0d5d9ee48)
at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/optimizer/prep/prepjointree.c:614
#11 0x00007fd0d47c5014 in subquery_planner (glob=glob(at)entry=0x7fd0d5d9edb8,
parse=parse(at)entry=0x7fd0d5d30d48, parent_root=parent_root(at)entry=0x0,
hasRecursion=hasRecursion(at)entry=0 '\000', tuple_fraction=0,
subroot=subroot(at)entry=0x7fff52170908)
at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/optimizer/plan/planner.c:374
#12 0x00007fd0d47c5975 in standard_planner (parse=0x7fd0d5d30d48,
cursorOptions=0, boundParams=0x0) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/optimizer/plan/planner.c:229
#13 0x00007fd0d4848034 in pg_plan_query (querytree=<optimized out>,
cursorOptions=cursorOptions(at)entry=0, boundParams=boundParams(at)entry=0x0) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/tcop/postgres.c:809
#14 0x00007fd0d4848124 in pg_plan_queries
(querytrees=querytrees(at)entry=0x7fd0d5d30cf8,
cursorOptions=0, boundParams=boundParams(at)entry=0x0) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/tcop/postgres.c:868
#15 0x00007fd0d4929760 in BuildCachedPlan
(plansource=plansource(at)entry=0x7fd0d5d7d940,
qlist=0x7fd0d5d30cf8, qlist(at)entry=0x0, boundParams=boundParams(at)entry=0x0)
at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/utils/cache/plancache.c:951
#16 0x00007fd0d4929a98 in GetCachedPlan
(plansource=plansource(at)entry=0x7fd0d5d7d940,
boundParams=boundParams(at)entry=0x0, useResOwner=useResOwner(at)entry=0 '\000')
at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/utils/cache/plancache.c:1165
#17 0x00007fd0d48497ab in exec_bind_message (input_message=0x7fff52170be0)
at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/tcop/postgres.c:1774
#18 PostgresMain (argc=<optimized out>, argv=argv(at)entry=0x7fd0d5c8d950,
dbname=0x7fd0d5c8d840 "databasename", username=<optimized out>) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/tcop/postgres.c:4071
#19 0x00007fd0d45f239c in BackendRun (port=0x7fd0d5cd2c00) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/postmaster/postmaster.c:4159
#20 BackendStartup (port=0x7fd0d5cd2c00) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/postmaster/postmaster.c:3835
#21 ServerLoop () at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/postmaster/postmaster.c:1609
#22 0x00007fd0d47f18e1 in PostmasterMain (argc=5, argv=<optimized out>) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/postmaster/postmaster.c:1254
#23 0x00007fd0d45f30cd in main (argc=5, argv=0x7fd0d5c8c970) at
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/main/main.c:221

Thanks,
Michael J. Bommarito II, CEO
Bommarito Consulting, LLC
*Web:* http://www.bommaritollc.com
*Mobile:* +1 (646) 450-3387

On Tue, Jul 7, 2015 at 11:49 PM, Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
wrote:

> On Wed, Jul 8, 2015 at 10:22 AM, Michael Bommarito
> <michael(at)bommaritollc(dot)com> wrote:
> > I'll try to generate an isolated core dump and stack trace this weekend
> > when the system is not in use.
>
> That would be nice. I have let pgbench -C run for one hour with select
> * from pg_stat_activity running every second (\watch 1) in parallel
> but I could not reproduce the issue on HEAD.
> --
> Michael
>