Re: SegFault on 9.6.14

Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com> writes:

> On Mon, Jul 15, 2019 at 07:22:55PM -0500, Jerry Sievers wrote:
>
>>Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com> writes:
>>
>>> On Mon, Jul 15, 2019 at 06:48:05PM -0500, Jerry Sievers wrote:
>>>
>>>>Greetings Hackers.
>>>>
>>>>We have a reproduceable case of $subject that issues a backtrace such as
>>>>seen below.
>>>>
>>>>The query that I'd prefer to sanitize before sending is <30 lines of at
>>>>a glance, not terribly complex logic.
>>>>
>>>>It nonetheless dies hard after a few seconds of running and as expected,
>>>>results in an automatic all-backend restart.
>>>>
>>>>Please advise on how to proceed. Thanks!
>>>>
>>>>bt
>>>>#0 initscan (scan=scan(at)entry=0x55d7a7daa0b0, key=0x0, keep_startblock=keep_startblock(at)entry=1 '\001')
>>>> at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:233
>>>>#1 0x000055d7a72fa8d0 in heap_rescan (scan=0x55d7a7daa0b0, key=key(at)entry=0x0) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:1529
>>>>#2 0x000055d7a7451fef in ExecReScanSeqScan (node=node(at)entry=0x55d7a7d85100) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSeqscan.c:280
>>>>#3 0x000055d7a742d36e in ExecReScan (node=0x55d7a7d85100) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:158
>>>>#4 0x000055d7a7445d38 in ExecReScanGather (node=node(at)entry=0x55d7a7d84d30) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeGather.c:475
>>>>#5 0x000055d7a742d255 in ExecReScan (node=0x55d7a7d84d30) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:166
>>>>#6 0x000055d7a7448673 in ExecReScanHashJoin (node=node(at)entry=0x55d7a7d84110) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeHashjoin.c:1019
>>>>#7 0x000055d7a742d29e in ExecReScan (node=node(at)entry=0x55d7a7d84110) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:226
>>>><about 30 lines omitted>
>>>>
>>>
>>> Hmmm, that means it's crashing here:
>>>
>>> if (scan->rs_parallel != NULL)
>>> scan->rs_nblocks = scan->rs_parallel->phs_nblocks; <--- here
>>> else
>>> scan->rs_nblocks = RelationGetNumberOfBlocks(scan->rs_rd);
>>>
>>> But clearly, scan is valid (otherwise it'd crash on the if condition),
>>> and scan->rs_parallel must me non-NULL. Which probably means the pointer
>>> is (no longer) valid.
>>>
>>> Could it be that the rs_parallel DSM disappears on rescan, or something
>>> like that?
>>
>>No clue but something I just tried was to disable parallelism by setting
>>max_parallel_workers_per_gather to 0 and however the query has not
>>finished after a few minutes, there is no crash.
>>
>
> That might be a hint my rough analysis was somewhat correct. The
> question is whether the non-parallel plan does the same thing. Maybe it
> picks a plan that does not require rescans, or something like that.
>
>>Please advise.
>>
>
> It would be useful to see (a) exacution plan of the query, (b) full
> backtrace and (c) a bit of context for the place where it crashed.
>
> Something like (in gdb):
>
> bt full
> list
> p *scan

The p *scan did nothing unless I ran it first however my gdb $foo isn't
strong presently.

I'll need to sanitize the explain output but can do so ASAP and send it
along.

Thx!

$ gdb /usr/lib/postgresql/9.6/bin/postgres core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/postgresql/9.6/bin/postgres...Reading symbols from /usr/lib/debug/.build-id/04/6f55a5ce6ce05064edfc8feee61c6cb039d296.debug...done.
done.
[New LWP 31654]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: foo_eis_segfault: jsievers staging 10.220.22.26(57948) SELECT '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 initscan (scan=scan(at)entry=0x55d7a7daa0b0, key=0x0, keep_startblock=keep_startblock(at)entry=1 '\001')
at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:233
233 /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c: No such file or directory.
(gdb) p *scan
$1 = {rs_rd = 0x7fa6c6935a08, rs_snapshot = 0x55d7a7c2e630, rs_nkeys = 0, rs_key = 0x0, rs_bitmapscan = 0 '\000', rs_samplescan = 0 '\000', rs_pageatatime = 1 '\001',
rs_allow_strat = 1 '\001', rs_allow_sync = 1 '\001', rs_temp_snap = 1 '\001', rs_nblocks = 198714, rs_startblock = 1920300133, rs_numblocks = 4294967295, rs_strategy = 0x55d7a7daa6a0,
rs_syncscan = 1 '\001', rs_inited = 0 '\000', rs_ctup = {t_len = 114, t_self = {ip_blkid = {bi_hi = 0, bi_lo = 62879}, ip_posid = 77}, t_tableOid = 994804890, t_data = 0x0},
rs_cblock = 4294967295, rs_cbuf = 0, rs_parallel = 0x7fa673a54108, rs_cindex = 76, rs_ntuples = 77, rs_vistuples = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66,
67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 28255, 25711, 24421, 25705, 12576, 8247, 29754, 29281, 25959, 27764, 29545, 8308, 31528, 16724, 18258, 21573, 20037, 21076,
8281, 25914, 28792, 8306, 22139, 21057, 14880, 24950, 28274, 8303, 12337, 14880, 24950, 24946, 29812, 28526, 13088, 14880, 24950, 29810, 28793, 8293, 13106, 14880, 24950, 29810, 28793,
28525, 8292, 12589, 14880, 24950, 25458, 27759, 26988, 8292, 8240, 30266, 29281, 25964, 25974, 29548, 28789, 12320, 14880, 24950, 28274, 28527, 25708, 12576, 8240, 30266, 29281, 24943,
29812, 28526, 13088, 14880, 28524, 24931, 26996, 28271, 13344, 13110, 8317, 29242, 29541, 28526, 12576, 14880, 25970, 28275, 28001, 8293, 15932, 14880, 25970, 29555, 29295, 26484,
28530, 28789, 25970, 8294, 8240, 29242, 29541, 29295, 26473, 25204, 8300, 8240, 29242, 29541, 29295, 26473, 28515, 8300, 8240, 29242, 29541...}}
(gdb) bt full
#0 initscan (scan=scan(at)entry=0x55d7a7daa0b0, key=0x0, keep_startblock=keep_startblock(at)entry=1 '\001')
at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:233
allow_strat = <optimized out>
allow_sync = <optimized out>
#1 0x000055d7a72fa8d0 in heap_rescan (scan=0x55d7a7daa0b0, key=key(at)entry=0x0) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:1529
__func__ = "heap_rescan"
#2 0x000055d7a7451fef in ExecReScanSeqScan (node=node(at)entry=0x55d7a7d85100) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSeqscan.c:280
scan = <optimized out>
#3 0x000055d7a742d36e in ExecReScan (node=0x55d7a7d85100) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:158
__func__ = "ExecReScan"
#4 0x000055d7a7445d38 in ExecReScanGather (node=node(at)entry=0x55d7a7d84d30) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeGather.c:475
No locals.
#5 0x000055d7a742d255 in ExecReScan (node=0x55d7a7d84d30) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:166
__func__ = "ExecReScan"
#6 0x000055d7a7448673 in ExecReScanHashJoin (node=node(at)entry=0x55d7a7d84110) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeHashjoin.c:1019
No locals.
#7 0x000055d7a742d29e in ExecReScan (node=node(at)entry=0x55d7a7d84110) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:226
__func__ = "ExecReScan"
#8 0x000055d7a7433ce7 in ExecProcNode (node=node(at)entry=0x55d7a7d84110) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:381
result = <optimized out>
__func__ = "ExecProcNode"
#9 0x000055d7a7452989 in ExecSort (node=node(at)entry=0x55d7a7d83ea0) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSort.c:103
plannode = <optimized out>
outerNode = 0x55d7a7d84110
tupDesc = <optimized out>
estate = 0x55d7a7d5fee8
dir = ForwardScanDirection
tuplesortstate = 0x55d7a7dd2448
slot = <optimized out>
#10 0x000055d7a7433de8 in ExecProcNode (node=0x55d7a7d83ea0) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:495
result = <optimized out>
__func__ = "ExecProcNode"
#11 0x000055d7a743ffe9 in fetch_input_tuple (aggstate=aggstate(at)entry=0x55d7a7d83528) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:598
slot = <optimized out>
#12 0x000055d7a7441bb3 in agg_retrieve_direct (aggstate=0x55d7a7d83528) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:2078
econtext = 0x55d7a7d838b0
pergroup = 0x55d7a7d8e758
firstSlot = 0x55d7a7d83960
numGroupingSets = 1
node = 0x7fa6c68a5da8
tmpcontext = 0x55d7a7d83750
peragg = 0x55d7a7d8d6b8
outerslot = <optimized out>
nextSetSize = <optimized out>
result = <optimized out>
hasGroupingSets = 0 '\000'
currentSet = <optimized out>
numReset = 1
i = <optimized out>
---Type <return> to continue, or q <return> to quit---
#13 ExecAgg (node=node(at)entry=0x55d7a7d83528) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:1903
result = <optimized out>
#14 0x000055d7a7433dc8 in ExecProcNode (node=node(at)entry=0x55d7a7d83528) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:503
result = <optimized out>
__func__ = "ExecProcNode"
#15 0x000055d7a744af74 in ExecLimit (node=node(at)entry=0x55d7a7d83288) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeLimit.c:91
direction = ForwardScanDirection
slot = <optimized out>
outerPlan = 0x55d7a7d83528
__func__ = "ExecLimit"
#16 0x000055d7a7433d28 in ExecProcNode (node=node(at)entry=0x55d7a7d83288) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:531
result = <optimized out>
__func__ = "ExecProcNode"
#17 0x000055d7a744ff69 in ExecNestLoop (node=node(at)entry=0x55d7a7d60cd0) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeNestloop.c:174
nl = 0x7fa6c68a6048
innerPlan = 0x55d7a7d83288
outerPlan = 0x55d7a7d610c0
outerTupleSlot = <optimized out>
innerTupleSlot = <optimized out>
joinqual = 0x0
otherqual = 0x0
econtext = 0x55d7a7d60de0
lc = <optimized out>
#18 0x000055d7a7433e28 in ExecProcNode (node=node(at)entry=0x55d7a7d60cd0) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:476
result = <optimized out>
__func__ = "ExecProcNode"
#19 0x000055d7a7452989 in ExecSort (node=node(at)entry=0x55d7a7d60a60) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSort.c:103
plannode = <optimized out>
outerNode = 0x55d7a7d60cd0
tupDesc = <optimized out>
estate = 0x55d7a7d5fee8
dir = ForwardScanDirection
tuplesortstate = 0x55d7a7d98398
slot = <optimized out>
#20 0x000055d7a7433de8 in ExecProcNode (node=0x55d7a7d60a60) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:495
result = <optimized out>
__func__ = "ExecProcNode"
#21 0x000055d7a743ffe9 in fetch_input_tuple (aggstate=aggstate(at)entry=0x55d7a7d60088) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:598
slot = <optimized out>
#22 0x000055d7a7441bb3 in agg_retrieve_direct (aggstate=0x55d7a7d60088) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:2078
econtext = 0x55d7a7d60440
pergroup = 0x55d7a7d91230
firstSlot = 0x55d7a7d604f0
numGroupingSets = 1
node = 0x7fa6c68a6328
tmpcontext = 0x55d7a7d602b0
peragg = 0x55d7a7d90190
outerslot = <optimized out>
nextSetSize = <optimized out>
---Type <return> to continue, or q <return> to quit---
result = <optimized out>
hasGroupingSets = 0 '\000'
currentSet = <optimized out>
numReset = 1
i = <optimized out>
#23 ExecAgg (node=node(at)entry=0x55d7a7d60088) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:1903
result = <optimized out>
#24 0x000055d7a7433dc8 in ExecProcNode (node=node(at)entry=0x55d7a7d60088) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:503
result = <optimized out>
__func__ = "ExecProcNode"
#25 0x000055d7a742ff2e in ExecutePlan (dest=0x7fa673a96308, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, operation=CMD_SELECT, use_parallel_mode=<optimized out>,
planstate=0x55d7a7d60088, estate=0x55d7a7d5fee8) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execMain.c:1567
slot = <optimized out>
current_tuple_count = 0
#26 standard_ExecutorRun (queryDesc=0x55d7a7d54718, direction=<optimized out>, count=0) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execMain.c:339
estate = 0x55d7a7d5fee8
operation = CMD_SELECT
dest = 0x7fa673a96308
sendTuples = <optimized out>
#27 0x00007fa6c7027515 in explain_ExecutorRun (queryDesc=0x55d7a7d54718, direction=ForwardScanDirection, count=0)
at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../contrib/auto_explain/auto_explain.c:281
save_exception_stack = 0x7fff4aeeaa80
save_context_stack = 0x0
local_sigjmp_buf = {{__jmpbuf = {94384722106264, 8229023444991490729, 0, 94384722102040, 0, 1, 8229023444890827433, 8250672449167702697}, __mask_was_saved = 0, __saved_mask = {
__val = {94384721739856, 140734450543072, 94384714940022, 140354273004312, 140354273004312, 140734450543104, 94384714691234, 2, 2, 140734450543200, 94384711690034, 2,
3462443396, 8388608, 3547611511646930944, 140734450543200}}}}
#28 0x00007fa6c6e1fdb0 in pgss_ExecutorRun (queryDesc=0x55d7a7d54718, direction=ForwardScanDirection, count=0)
at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../contrib/pg_stat_statements/pg_stat_statements.c:875
save_exception_stack = 0x7fff4aeeac20
save_context_stack = 0x0
local_sigjmp_buf = {{__jmpbuf = {94384722106264, 8229023444960033449, 0, 94384722102040, 0, 1, 8229023444993587881, 8250670555334589097}, __mask_was_saved = 0, __saved_mask = {
__val = {4294967296, 140354272256808, 94384714928429, 16, 94384719269552, 24, 94384720895528, 94384722102040, 0, 140734450543408, 94384714928429, 94384722106264,
94384720895528, 140734450543440, 94384714994982, 94384722106264}}}}
#29 0x000055d7a7553167 in PortalRunSelect (portal=portal(at)entry=0x55d7a7d55798, forward=forward(at)entry=1 '\001', count=0, count(at)entry=9223372036854775807, dest=dest(at)entry=0x7fa673a96308)
at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/tcop/pquery.c:948
queryDesc = 0x55d7a7d54718
direction = <optimized out>
nprocessed = <optimized out>
__func__ = "PortalRunSelect"
#30 0x000055d7a75547a0 in PortalRun (portal=portal(at)entry=0x55d7a7d55798, count=count(at)entry=9223372036854775807, isTopLevel=isTopLevel(at)entry=1 '\001', dest=dest(at)entry=0x7fa673a96308,
altdest=altdest(at)entry=0x7fa673a96308, completionTag=completionTag(at)entry=0x7fff4aeeb050 "") at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/tcop/pquery.c:789
save_exception_stack = 0x7fff4aeeaf00
save_context_stack = 0x0
local_sigjmp_buf = {{__jmpbuf = {94384721085312, 8229023445033433769, 94384722106264, 140352881779464, 94384721085584, 2, 8229023444955839145, 2765920793019169449},
__mask_was_saved = 0, __saved_mask = {__val = {0, 12099560782865280144, 0, 8, 8, 140734450544226, 1, 88, 94384722106264, 94384715935530, 94384721085584, 140734450543840,
94384714930017, 2, 94384722106264, 140734450543872}}}}
result = <optimized out>
nprocessed = <optimized out>
saveTopTransactionResourceOwner = 0x55d7a7c118e8
---Type <return> to continue, or q <return> to quit---
saveTopTransactionContext = 0x55d7a7c10eb8
saveActivePortal = 0x0
saveResourceOwner = 0x55d7a7c118e8
savePortalContext = 0x0
saveMemoryContext = 0x55d7a7c10eb8
__func__ = "PortalRun"
#31 0x000055d7a75512d6 in exec_simple_query (
query_string=0x55d7a7ce6b38 "select v.account_id, COUNT(cnt.clicks), te.description,\nl.product_id\nfrom nbox_nc_ah.tracking_events te\njoin nbox_nc_ah.page_views pv on pv.page_view_id = te.page_view_id\njoin nbox_nc_ah.visits v on v"...) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/tcop/postgres.c:1109
parsetree = 0x55d7a7c5c380
portal = 0x55d7a7d55798
snapshot_set = <optimized out>
commandTag = <optimized out>
completionTag = "\000\370\253\247\327U\000\000\240F\252\247\327U\000\000\200\260\356J\377\177\000\000\215\326g\247\327U\000\000\300\260\356J\377\177\000\000d\261\356J\377\177\000\000\240\260\356J\377\177\000\000v\031F\247\327U\000"
querytree_list = <optimized out>
plantree_list = 0x7fa673a962d8
receiver = 0x7fa673a96308
format = 0
dest = DestRemote
parsetree_list = 0x55d7a7c5c4b0
save_log_statement_stats = 0 '\000'
was_logged = 0 '\000'
msec_str = "\020\261\356J\377\177\000\000(\002", '\000' <repeats 14 times>, "\340?\256\247\327U\000"
parsetree_item = 0x55d7a7c5c490
isTopLevel = 1 '\001'
#32 PostgresMain (argc=<optimized out>, argv=argv(at)entry=0x55d7a7c56830, dbname=0x55d7a7c11b88 "staging", username=<optimized out>)
at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/tcop/postgres.c:4101
query_string = 0x55d7a7ce6b38 "select v.account_id, COUNT(cnt.clicks), te.description,\nl.product_id\nfrom nbox_nc_ah.tracking_events te\njoin nbox_nc_ah.page_views pv on pv.page_view_id = te.page_view_id\njoin nbox_nc_ah.visits v on v"...
firstchar = -1479190632
input_message = {
data = 0x55d7a7ce6b38 "select v.account_id, COUNT(cnt.clicks), te.description,\nl.product_id\nfrom nbox_nc_ah.tracking_events te\njoin nbox_nc_ah.page_views pv on pv.page_view_id = te.page_view_id\njoin nbox_nc_ah.visits v on v"..., len = 1042, maxlen = 2048, cursor = 1042}
local_sigjmp_buf = {{__jmpbuf = {140734450544288, 8229023445169748649, 94384721061936, 1, 94384721061720, 94384721052928, 8229023445035530921, 2765920790734322345},
__mask_was_saved = 1, __saved_mask = {__val = {0, 94386201296895, 94384713689589, 18446603339259007057, 140354407146656, 0, 1305670059009, 32, 4, 489626271867, 0, 0,
532575944823, 140734450544608, 0, 140734450544704}}}}
send_ready_for_query = 0 '\000'
disable_idle_in_transaction_timeout = <optimized out>
__func__ = "PostgresMain"
#33 0x000055d7a72c6a1b in BackendRun (port=0x55d7a7c54500) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/postmaster/postmaster.c:4339
ac = 1
secs = 616545808
usecs = 503344
i = 1
av = 0x55d7a7c56830
maxac = <optimized out>
#34 BackendStartup (port=0x55d7a7c54500) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/postmaster/postmaster.c:4013
bn = <optimized out>
---Type <return> to continue, or q <return> to quit---
pid = <optimized out>
#35 ServerLoop () at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/postmaster/postmaster.c:1722
rmask = {fds_bits = {16, 0 <repeats 15 times>}}
selres = <optimized out>
now = <optimized out>
readmask = {fds_bits = {48, 0 <repeats 15 times>}}
last_lockfile_recheck_time = 1563230588
last_touch_time = 1563230588
__func__ = "ServerLoop"
#36 0x000055d7a74ed281 in PostmasterMain (argc=13, argv=<optimized out>) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/postmaster/postmaster.c:1330
opt = <optimized out>
status = <optimized out>
userDoption = <optimized out>
listen_addr_saved = 1 '\001'
i = <optimized out>
output_config_variable = <optimized out>
__func__ = "PostmasterMain"
#37 0x000055d7a72c7bf1 in main (argc=13, argv=0x55d7a7c0f840) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/main/main.c:228
No locals.
(gdb) list
228 in /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c
(gdb)

>
>
>
> regards

--
Jerry Sievers
Postgres DBA/Development Consulting
e: postgres(dot)consulting(at)comcast(dot)net