Re: segmentation fault postgres 9.3.5 core dump perlu related ?

Hi Alan,

Thanks for your input.

My initial simplistic stress test ( two connections calling same suspect function in a loop ) has failed in causing the problem albeit I have not used any range of inputs for the possible parameters. Given your thoughts on the the internal mechnanics it seems unlikely it is competing sessions. I’ll see about varying and logging arguments in future testing. Reproducing is 90 % of the battle and
unfortunately we are losing on that front currently.

When I type (gdb) info threads on the most recent core file I see:
* 1 Thread 802c06400 (LWP 101353/postgres) 0x00000000005756b8 in ExecProcNode ()
Not sure that fits with your expectations.

We only have two invoked perl functions in the database both of which are plperlu. These functions are
both invoked at least once in a normal usage scenario, which makes the infrequency of the segmentation fault puzzling.

Regards

Dave

From: Alex Hunsaker [mailto:badalex(at)gmail(dot)com]
Sent: Thursday, January 29, 2015 12:58 AM
To: Day, David
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: [GENERAL] segmentation fault postgres 9.3.5 core dump perlu related ?

On Wed, Jan 28, 2015 at 1:23 PM, Day, David <dday(at)redcom(dot)com<mailto:dday(at)redcom(dot)com>> wrote:
It has been some time since we have seen this problem.
See earlier message on this subject/thread for the suspect plperl function executing
at the time of the core.

Someone on our development team suggested it might relate to some build options of perl.
In particular MULTIPLICITY or THREADS . We can have this perl fx executing on
two different connections/sessions at the same time.

Hrm, I can't see how >1 connections/sessions could tickle the bug. Or THREADS/MULTIPLICITY, short of some perl bug. Each backend is its own process and so each perl interpreter is isolated at from each other at that level. IOW each new connection has its very own perl interpreter that has no shared state with any of the others (short of using $_SHARED). But hey, if your testing finds it is easier to trigger with more connections, it just makes the bug more interesting :).

open as use use it should just be standard pipe(); fork(); exec(); dance. And I'm fairly certain perl does not do anything magic like making a thread behind the scene. In gdb you could also try "info threads", just to see if somehow a thread did created.

Multiplicity should only come into play if you use plperl and plperlu in the same session (without it, it should error out with "Cannot allocate multiple Perl interpreters on this platform").

I believe below is an valid stack dump:

Core was generated by `postgres'.
Program terminated with signal 11, Segmentation fault.
(gdb) bt
#0 0x000000080bfa50a3 in Perl_fbm_instr () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18
#1 0x000000080c00ff93 in Perl_re_intuit_start () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18
#2 0x000000080bfc27a2 in Perl_pp_match () from /usr/local/lib/perl5/5.18/mach/CORE/libperl.so.5.18

This sure makes it look like it is segfaulting on some kind of regex /not/ open.

Any chance you could come up with a reproducible test case? I suspect the inputs to the function might help narrow it down to something reproducible. Maybe log the arguments at the start of the function? Or perhaps in your middleware when calling the function crashes, log how it was called?