Re: Recovery of PGSQL after system crash failing!!!

On Sun, 11 Feb 2001, Vadim Mikheev wrote:

> Please try to restart with option wal_debug = 1 so postmaster log
> will be more informative and send this log me.

I enabled 'wal_debug=1' via both the -c command line option and
(seperately) via ./data/postgresql.conf, as well as setting wal_debug=16
in ./data/postgresql.conf and I got no addition postmaster log information
than in my last email. :(
Also set my coredump limit to unlimited (ulimit -c unlimited) and
started postmaster up. I got a core file, and here is what gdb has to say
about it:

GNU gdb 19990928
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
(no debugging symbols found)...
Core was generated by `postmaster -d 5 -D /usr/local/pgsql/data/'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/libreadline.so.4...(no debugging symbols found)...done.
Reading symbols from /lib/libncurses.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols
found)...done.
#0 0x20c931 in kill () from /lib/libc.so.6
(gdb) bt
#0 0x20c931 in kill () from /lib/libc.so.6
#1 0x20c618 in raise () from /lib/libc.so.6
#2 0x20dc71 in abort () from /lib/libc.so.6
#3 0x8080495 in XLogFileOpen ()
#4 0x8080b52 in ReadRecord ()
#5 0x8081f66 in StartupXLOG ()
#6 0x80853ea in BootstrapMain ()
#7 0x80ee1e7 in SSDataBase ()
#8 0x80ec766 in PostmasterMain ()
#9 0x80cd194 in main ()
#10 0x206a42 in __libc_start_main () from /lib/libc.so.6

Also, since it appears it died in XLogFileOpen(), here is what the
directory structure looks like for xlog related files:

drwx--S--- 5 postgres postgres 4096 Feb 12 20:51 data
drwx--S--- 2 postgres postgres 4096 Feb 11 04:12 data/pg_xlog
-rw------- 1 postgres postgres 16777216 Feb 11 04:12 data/pg_xlog/0000000000000030

The file listed in data/pg_xlog is the only file in this directory. Does
not look like a lot of help to me, but here it is also.
One other wrench to thrown into the works... The kernel on this
machine is 2.2.18 with the patches listed at www.linuxraid.org applied. I
have a feeling that the linux-security patches mentioned on that page may
be giving pgsql heartburn on recovery. I am going to recompile the kernel
w/o them enabled and see if anything different results, and will post my
results.

> > data; initdb'? A quick response would be greatly appreciated. Thanks.
>
> Please archieve PG' data dir - it probably will be useful to find bug.

Archived. It is a bit over 11MB, and I can put it on my web server
if some one wants to look at it (10 minute download with a 192kbit or
faster link). Though I would like to limit its distribution as it does
have relatively sensitive company data buried in it (custom lists and the
like).
Though there is nothing I need to retrieve from it... This
database is from the web site that is updated every night from the
internal databases. For the time being I have fallen back to 7.0.3 for
production use.
Thank you for all of your help. TTYL.

---------------------------------------------------------------------------
| "For to me to live is Christ, and to die is gain." |
| --- Philippians 1:21 (KJV) |
---------------------------------------------------------------------------
| Ryan Kirkpatrick | Boulder, Colorado | http://www.rkirkpat.net/ |
---------------------------------------------------------------------------