Re: Disaster!

pg_clog information:

# cd pg_clog
# ls -al
total 3602
drwx------ 2 pgsql pgsql 512 Jan 23 03:49 .
drwx------ 6 pgsql pgsql 512 Jan 23 12:30 ..
-rw------- 1 pgsql pgsql 262144 Jan 18 19:43 0000
-rw------- 1 pgsql pgsql 262144 Jan 18 19:43 0001
-rw------- 1 pgsql pgsql 262144 Jan 18 19:43 0002
-rw------- 1 pgsql pgsql 262144 Jan 18 19:43 0003
-rw------- 1 pgsql pgsql 262144 Jan 19 08:35 0004
-rw------- 1 pgsql pgsql 262144 Jan 21 10:35 0005
-rw------- 1 pgsql pgsql 262144 Jan 20 10:07 0006
-rw------- 1 pgsql pgsql 262144 Jan 21 19:30 0007
-rw------- 1 pgsql pgsql 262144 Jan 21 17:24 0008
-rw------- 1 pgsql pgsql 262144 Jan 22 06:50 0009
-rw------- 1 pgsql pgsql 262144 Jan 22 13:01 000A
-rw------- 1 pgsql pgsql 262144 Jan 23 06:45 000B
-rw------- 1 pgsql pgsql 262144 Jan 23 09:37 000C
-rw------- 1 pgsql pgsql 163840 Jan 23 12:30 000D

I don't have debug symbols, so backtrace isn't that helpful:

(gdb) bt
#0 0x2840a848 in kill () from /usr/lib/libc.so.4
#1 0x2844ee90 in abort () from /usr/lib/libc.so.4
#2 0x81b33ba in ?? ()
#3 0x8092a0c in ?? ()
#4 0x8092450 in ?? ()
#5 0x808a9d7 in ?? ()
#6 0x808adf4 in ?? ()
#7 0x808ae8c in ?? ()
#8 0x808bffa in ?? ()
#9 0x809082f in ?? ()
#10 0x8095204 in ?? ()
#11 0x8135c6b in ?? ()
#12 0x813309c in ?? ()
#13 0x8108dc3 in ?? ()
#14 0x806d49b in ?? ()

I have a backup of the data dir for future analysis, but this is kind of
an urgent fix right now! (Especially since it's 4am my time :( )

Thanks,

Chris

Christopher Kings-Lynne wrote:

> We ran out of disk space on our main server, and now I've freed up
> space, we cannot start postgres!
>
> Jan 23 12:18:50 canaveral postgres[563]: [2-1] LOG: checkpoint record
> is at 2/96500B94
> Jan 23 12:18:50 canaveral postgres[563]: [3-1] LOG: redo record is at
> 2/964BD23C; undo record is at 0/0; shutdown FALSE
> Jan 23 12:18:50 canaveral postgres[563]: [4-1] LOG: next transaction
> ID: 14216463; next OID: 4732327
> Jan 23 12:18:50 canaveral postgres[563]: [5-1] LOG: database system was
> not properly shut down; automatic recovery in progress
> Jan 23 12:18:50 canaveral postgres[563]: [6-1] LOG: redo starts at
> 2/964BD23C
> Jan 23 12:18:51 canaveral postgres[563]: [7-1] PANIC: could not access
> status of transaction 14286850
> Jan 23 12:18:51 canaveral postgres[563]: [7-2] DETAIL: could not read
> from file "/usr/local/pgsql/data/pg_clog/000D" at offset 163840:
> Undefined error: 0
> Jan 23 12:18:51 canaveral postgres[567]: [1-1] FATAL: the database
> system is starting up
> Jan 23 12:18:52 canaveral postgres[558]: [1-1] LOG: startup process
> (PID 563) was terminated by signal 6
> Jan 23 12:18:52 canaveral postgres[558]: [2-1] LOG: aborting startup
> due to startup process failure
>
> What can I do?
>
> Chris
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
> joining column's datatypes do not match