Re: segmentation fault postgres 9.3.5 core dump perlu related ?

Hi,

Update : A storey with a happy ending.

I have not seen this segmentation fault since converting the pgperl functions to python within the
FreeBSD 9.x environment. So I believe Guy Helmer’s suggested causation was likely spot on.
Due to the inability to reproduce the issue on demand there is a small chance this is not the root cause, but I’ll let the current empirical health of the system speak loudest on this matter.

We are in the process of migrating development efforts to 10.x so selection of perl over python should become a non-issue.

Thanks all who assisted me in figuring this out.

Best Regards

Dave

From: Day, David
Sent: Wednesday, February 18, 2015 8:07 AM
To: 'Guy Helmer'
Cc: 'pgsql-general(at)postgresql(dot)org'
Subject: RE: [GENERAL] segmentation fault postgres 9.3.5 core dump perlu related ?

Update/Information sharing: ( FreeBSD 10.0 (amd64) – Postgres 9.3.5 – Perl 5.18 )

I have converted our Postgres plperlu functions to plpython2u to see if the postgres segmentation faults disappear.
Lacking a known way to reproduce the error on demand, I will have to wait a few weeks for the absence of the symptom before I might conclude that this bug reported to me by Guy Helmer was the issue. Migration/Upgrade to FreeBsd 10.1 was not an immediate option.

Regards

Dave

----
Guy,

No I had not seen that bug report before. ( https://rt.perl.org/Public/Bug/Display.html?id=122199 )

We did migrate from FreeBSD 9.x (2?) and I think it true
that we were not experiencing the problem at time.
So it might be a good fit/explanation for our current experience

There were a couple of suggestions to follow up on.
I’ll keep the thread updated.

Thanks, a good start to my Friday the 13th.

Regards

Dave Day

From: Guy Helmer [mailto:ghelmer(at)palisadesystems(dot)com]
Sent: Thursday, February 12, 2015 6:19 PM
To: Day, David
Cc: pgsql-general(at)postgresql(dot)org<mailto:pgsql-general(at)postgresql(dot)org>
Subject: Re: [GENERAL] segmentation fault postgres 9.3.5 core dump perlu related ?

On Feb 12, 2015, at 3:21 PM, Day, David <dday(at)redcom(dot)com<mailto:dday(at)redcom(dot)com>> wrote:

Update/Information sharing on my pursuit of segmentation faults

FreeBSD 10.0-RELEASE-p12 amd64
Postgres version 9.3.5

Below are three postgres core files generated from two different machine ( Georgia and Alabama ) on Feb 11.
These cores would not be caused from an environment update issue that I last suspected might be causing the segfaults
So I am kind of back to square one in terms of thinking what is occurring.

? I am not sure that I understand the associated time events in the postgres log file output. Is this whatever happens to be running on the other postgress forked process when the cored process was detected ?
If this is the case then I have probably been reading to much from the content of the postgres log file at the time of core.
This probably just represents collateral damage of routine transactions that were in other forked processes at the time one of the processes cored ?

Therefore I would now just assert that postgres has a sporadic segmentation problem, no known way to reliably cause it
and am uncertain as to how to proceed to resolve it.

. . .

Georgia-Core 8:38 - Feb 11
[New process 101032]
[New Thread 802c06400 (LWP 101032)]
Core was generated by `postgres'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000080c4b6d51 in Perl_hfree_next_entry () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18
(gdb) bt
#0 0x000000080c4b6d51 in Perl_hfree_next_entry () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18
#1 0x000000080c4cab49 in Perl_sv_clear () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18
#2 0x000000080c4cb13a in Perl_sv_free2 () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18
#3 0x000000080c4e5102 in Perl_free_tmps () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18
#4 0x000000080bcfedea in plperl_destroy_interp () from /usr/local/lib/postgresql/plperl.so
#5 0x000000080bcfec05 in plperl_fini () from /usr/local/lib/postgresql/plperl.so
#6 0x00000000006292c6 in ?? ()
#7 0x000000000062918d in proc_exit ()
#8 0x00000000006443f3 in PostgresMain ()
#9 0x00000000005ff267 in PostmasterMain ()
#10 0x00000000005a31ba in main ()
(gdb) info threads
Id Target Id Frame
* 2 Thread 802c06400 (LWP 101032) 0x000000080c4b6d51 in Perl_hfree_next_entry () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18
* 1 Thread 802c06400 (LWP 101032) 0x000000080c4b6d51 in Perl_hfree_next_entry () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18

Given two of the coredumps are in down in libperl and this is FreeBSD 10.0 amd64, have you seen this?

https://rt.perl.org/Public/Bug/Display.html?id=122199

Michael Moll suggested trying setting vm.pmap.pcid_enabled to 0 but I don’t recall seeing if that helped.

Guy