Re: (Never?) Kill Postmaster?

=?ISO-8859-1?Q?Christian_Schr=F6der?= <cs(at)deriva(dot)de> writes:
> db2:/home/pgsql/data # gdb --pid=7129
> GNU gdb 6.5
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "x86_64-suse-linux".
> Attaching to process 7129
> Reading symbols from /usr/local/pgsql_8.2.5/bin/postgres...done.
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Reading symbols from /lib64/libcrypt.so.1...done.
> Loaded symbols for /lib64/libcrypt.so.1
> Reading symbols from /lib64/libdl.so.2...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libm.so.6...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /usr/local/pgsql_8.2.5/lib/plpgsql.so...done.
> Loaded symbols for /usr/local/pgsql_8.2.5/lib/plpgsql.so
> Reading symbols from /usr/local/pgsql_8.2.5/lib/plperl.so...done.
> Loaded symbols for /usr/local/pgsql_8.2.5/lib/plperl.so
> Reading symbols from
> /usr/lib/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so...done.
> Loaded symbols for
> /usr/lib/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
> Reading symbols from /lib64/libpthread.so.0...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 47248855881456 (LWP 7129)]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from
> /usr/lib/perl5/5.8.8/x86_64-linux-thread-multi/auto/Opcode/Opcode.so...done.
> Loaded symbols for
> /usr/lib/perl5/5.8.8/x86_64-linux-thread-multi/auto/Opcode/Opcode.so
> 0x00002af904809a68 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
> (gdb) bt
> #0 0x00002af904809a68 in __lll_mutex_lock_wait () from
> /lib64/libpthread.so.0
> #1 0x00002af904806e88 in pthread_rwlock_rdlock () from
> /lib64/libpthread.so.0
> #2 0x00002af8fb13de23 in _nl_find_msg () from /lib64/libc.so.6
> #3 0x00002af8fb13ec83 in __dcigettext () from /lib64/libc.so.6
> #4 0x00002af8fb186f0b in strerror_r () from /lib64/libc.so.6
> #5 0x00002af8fb186d33 in strerror () from /lib64/libc.so.6
> #6 0x00000000005f4daa in expand_fmt_string ()
> #7 0x00000000005f6d14 in errmsg ()
> #8 0x00000000005182cc in internal_flush ()
> #9 0x00000000005183b6 in internal_putbytes ()
> #10 0x000000000051841c in pq_putmessage ()
> #11 0x00000000005199c4 in pq_endmessage ()
> #12 0x0000000000440c6a in printtup ()
> #13 0x00000000004fc1b8 in ExecutorRun ()
> #14 0x0000000000580451 in PortalRunSelect ()
> #15 0x0000000000581446 in PortalRun ()
> #16 0x000000000057d625 in exec_simple_query ()
> #17 0x000000000057ea72 in PostgresMain ()
> #18 0x0000000000558218 in ServerLoop ()
> #19 0x0000000000558db8 in PostmasterMain ()
> #20 0x000000000051a213 in main ()

Too bad you don't have debug symbols :-(. However, this does raise a
couple of interesting questions:

* The only place internal_flush would call errmsg is here:

ereport(COMMERROR,
(errcode_for_socket_access(),
errmsg("could not send data to client: %m")));

So why is it unable to send data to the client?

* How the heck would strerror() be blocking on a thread mutex, when
the backend is single-threaded?

We recently discovered that it was possible for pltcl to cause the
backend to become multithreaded:
http://archives.postgresql.org/pgsql-patches/2007-09/msg00194.php
I see from your dump that this session has been using plperl, so I
wonder whether plperl has the same sort of issue. Can you determine
exactly what's been done with plperl in this session?

Also, can you confirm that there is actually more than one thread active
in this process? On Linux "ps -Lw" would show threads.

regards, tom lane