Re: random backend crashes - how to debug ( Is crash dump handler released ? )

On 15/06/2011 7:50 AM, Craig Ringer wrote:
> There's no reason it can't be compiled for PostgreSQL 8.4, though I
> never tested that. It shouldn't take long so I'll give it a go and get
> back to you.

Okies. I've built a version for 8.4.

You can download it (32-bit only) from:

http://www.postnewspapers.com.au/~craig/webfiles/crashdump_pg_84_32bit/crashdump.dll

If you're on Windows XP or Vista you will also need:

http://www.postnewspapers.com.au/~craig/webfiles/crashdump_pg_84_32bit/dbghelp.dll

... and since this DLL was compiled with VC++ 2008, you'll need the
VC++2008 redist installed if you don't already have it:

http://www.microsoft.com/downloads/en/details.aspx?familyid=9b2da534-3e03-4391-8a4d-074b9f2bc1bf&displaylang=en

Put crashdump.dll and dbghelp.dll into
C:\Program Files\PostgreSQL\8.4\lib

Edit postgresql.conf, uncomment shared_preload_libraries if it's
commented out and add 'crashdump' to it, eg:

shared_preload_libraries = 'crashdump'

Create a folder called "crashdumps" inside the data directory, at the
same level as the "pg_log", "pg_xlog", "base" etc directories. Get
properties on the new "crashdumps" directory and in the security tab add
"Full Control" to the "postgres" user. Save your changes.

Stop and start the postgresql-8.4 service from Start->Run->services.msc.

You should now have a working crash dump handler. To test it, run:

CREATE FUNCTION crashdump_crashme() RETURNS void AS 'crashdump.dll'
LANGUAGE 'C';

then invoke it to crash your database system:

SELECT crashdump_crashme();

If all goes well (heh) you'll lose your connection and the server will
crash and - hopefully - restart. If it doesn't restart, relaunch it
manually using services.msc.

You should now see a file in the "crashdumps" folder.You can email it to
me directly and I'll extract a backtrace.

Alternately, if you want to get the backtrace yourself you will need to
set up your NT_SYMBOL_PATH environment variable to match your install as
per the instructions here:

http://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Windows#Configuring_the_symbol_path

You will then be able to open it for debugging using Microsoft Visual
Studio 2008 Express Edition (or any paid Visual Studio edition). Once
open, right click on the dump file in the left bar and choose "debug new
instance". Alternately you can use windbg.exe from Debugging Tools for
Windows to analyse the dump as per the instructions here:

http://archives.postgresql.org/message-id/4CAB4294.2070104@postnewspapers.com.au

You'll probably want to

DROP FUNCTION crashdump_crashme();

after running your crashme test, though it's harmless if left in place
so long as it's not invoked. Dropping the crashme function doesn't
affect the crashdump handler; it's loaded by shared_preload_libraries
and will remain in place.

--
Craig Ringer

Tech-related writing at http://soapyfrogs.blogspot.com/