Re: Segmentation Fault

And now it's happening again...

I'm trying to get a core dump but I can't seem to actually generate one.

When I try to set ulimit -c from the command line, I get a message that I'm
not allowed to do that. So I went in and updated /etc/security/limits.conf
for the postgres user as follows.
postgres hard core unlimited
postgres soft core unlimited

I then restarted the postgres server. When I look at the limits in the pid
folder I see...
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes

My core_pattern is set to "/tmp/core-%e-%s-%u-%g-%p-%t"

I run the query that forces the crash but there is no core file in any of
/tmp/
/etc/postgresql/15
/etc/postgresql/15/main
/var/lib/postgresql/15
/var/lib/postgresql/15/main
/usr/lib/postgresql/15
/usr/lib/postgresql/15/lib
/usr/lib/postgresql/15/bin

GDB (more below) is even able to detect that there's a segmentation fault.

What am I doing wrong? Why can't I get a core file anywhere?

rjsjr

From the GDB shell... which looks identical to what's in the log file...
(gdb) set pagination off
(gdb) set logging file /datadrive/pgdebug.txt
(gdb) set logging on
Warning: 'set logging on', an alias for the command 'set logging enabled',
is deprecated.
Use 'set logging enabled on'.

Copying output to /datadrive/pgdebug.txt.
Copying debug output to /datadrive/pgdebug.txt.
(gdb) b errfinish
Breakpoint 1 at 0xaaaabef843b8
(gdb) cont
Continuing.

Program received signal SIGUSR1, User defined signal 1.
0x0000fff7735c766c in ?? () from /lib/aarch64-linux-gnu/libLLVM-10.so.1
(gdb) bt
#0 0x0000fff7735c766c in ?? () from /lib/aarch64-linux-gnu/libLLVM-10.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) bt
#0 0x0000fff7735c766c in ?? () from /lib/aarch64-linux-gnu/libLLVM-10.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) cont
Continuing.

Program received signal SIGUSR1, User defined signal 1.
0x0000fff7735c766c in ?? () from /lib/aarch64-linux-gnu/libLLVM-10.so.1
(gdb) bt
#0 0x0000fff7735c766c in ?? () from /lib/aarch64-linux-gnu/libLLVM-10.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x0000fff7add7a000 in ?? ()
(gdb) bt
#0 0x0000fff7add7a000 in ?? ()
#1 0x0000aaaacc627ca0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) cont
Continuing.

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb)

On Thu, Sep 14, 2023 at 8:27 AM Robert Sanford <wobbet(at)gmail(dot)com> wrote:

> Woke up this morning excited to create a core dump and... it worked.
>
> Not only does it work, it works fast. Query time dropped from 5.5s to
> 1.2s. That was what I was hoping to get out of the VACUUM ANALYZE.
>
> Very confused. I don't get it. I hope it doesn't happen again, but if it
> does I'll get the core dump.
>
> rjsjr
>
>
> On Wed, Sep 13, 2023 at 6:58 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
>> Robert Sanford <wobbet(at)gmail(dot)com> writes:
>> > I realized that when I was running GDB I just took a snap of when it
>> > failed. Here I've added after continuing and requesting a backtrace.
>>
>> This is still quite unhelpful; it appears to show normal behavior in
>> a process that's not the failing one. There is some advice about
>> collecting useful backtraces at
>>
>>
>> https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
>>
>> I'd counsel the method of enabling core dumps and then attaching
>> gdb to a core file, rather than dealing with live processes.
>>
>> regards, tom lane
>>
>