Re: Core dump

| > Sparc solaris 2.7 with postgres 7.0.2
| > It seems to be reproducable, the server crashes on us at a rate of about
| > every few hours.
|
| That's a very bizarre backtrace. Why the multiple levels of recursive
| entry to the quickdie() signal handler? I wonder if you aren't looking
| at some kind of Solaris bug --- perhaps it's not able to cope with a
| signal handler turning around and issuing new kernel calls.

I'm not sure that is the issue, see below.

| The core file you are looking at is probably *not* from the original
| failure, whatever that is. The sequence is probably
|
| 1. Some backend crashes for unknown reason, dumping core.
|
| 2. Postmaster observes messy death of a child, decides that mass suicide
| followed by restart is called for. Postmaster sends SIGUSR1 to all
| remaining backends to make them commit hara-kiri.
|
| 3. One or more other backends crash trying to obey postmaster's command.
| The corefile left for you to examine comes from whichever crashed
| last.
|
| So there are at least two problems here, but we only have evidence of
| the second one.
|
| Since the problem is fairly reproducible, I'd suggest you temporarily
| dike out the elog(NOTICE) call in quickdie() (in
| src/backend/tcop/postgres.c), which will probably allow the backends
| to honor SIGUSR1 without dumping core. Then you have a shot at seeing
| the core from the original failure.

I will try this, however the database is currently running under light load.
Only under high load does postgres start to choke, and eventually die.

| Assuming that this works (ie, you find a core that's not got anything
| to do with quickdie()), I'd suggest an inquiry to Sun about whether
| their signal handler logic hasn't got a problem with write() issued
| from inside a signal handler. Meanwhile let us know what the new
| backtrace shows.

I wrote a quick test program to test this theory. Below is the code and the
output.

#include <sys/types.h>
#include <stdio.h>
#include <unistd.h>
#include <signal.h>

static void moo (int);

int
main (void)
{
signal(SIGUSR1, moo);
raise(SIGUSR1);
}

static void
moo (cow)
int cow;
{
printf("Getting ready for write()\n");
write(STDOUT_FILENO, "Hello!\n", 7);
printf("Done.\n");
}

static void
moo (cow)
int cow;
{
printf("Getting ready for write()\n");
write(STDOUT_FILENO, "Hello!\n", 7);
printf("Done.\n");
}

eclipse% ./x
Getting ready for write()
Hello!
Done.
eclipse%

It would appear from that very rough test program that solaris doesn't mind
system calls from within a signal handler.

--
Man is a rational animal who always loses his temper when he is called
upon to act in accordance with the dictates of reason.
-- Oscar Wilde