Re: Out of memory (Failed on request size 24)

Breakpoint 1, 0x00000001000019dc in errfinish () from postmaster
(gdb) bt
#0 0x00000001000019dc in errfinish () from postmaster
#1 0x000000010000a680 in AllocSetAlloc () from postmaster
#2 0x0000000100002a1c in MemoryContextAlloc () from postmaster
#3 0x0000000100108c28 in _bt_search () from postmaster
#4 0x0000000100106484 in _bt_first () from postmaster
#5 0x00000001001045b4 in btgettuple () from postmaster
#6 0x0000000100029fb0 in FunctionCall2 () from postmaster
#7 0x00000001000295f8 in index_getnext () from postmaster
#8 0x000000010002942c in systable_getnext () from postmaster
#9 0x000000010000f9a0 in ScanPgRelation () from postmaster
#10 0x0000000100011088 in RelationBuildDesc () from postmaster
#11 0x000000010000e6fc in RelationSysNameGetRelation () from postmaster
#12 0x000000010000e620 in relation_openr () from postmaster
#13 0x000000010000e44c in heap_openr () from postmaster
#14 0x0000000100041044 in RelationBuildTriggers () from postmaster
#15 0x00000001000111a4 in RelationBuildDesc () from postmaster
#16 0x000000010000e6fc in RelationSysNameGetRelation () from postmaster
#17 0x000000010000e620 in relation_openr () from postmaster
#18 0x000000010000e44c in heap_openr () from postmaster
#19 0x000000010000e1c8 in CatalogCacheInitializeCache () from postmaster
#20 0x000000010000dab8 in SearchCatCache () from postmaster
#21 0x000000010000da3c in SearchSysCache () from postmaster
#22 0x000000010028b570 in InitializeSessionUserId () from postmaster
#23 0x0000000100288ae8 in InitPostgres () from postmaster
#24 0x000000010029e2a8 in PostgresMain () from postmaster
#25 0x00000001002990e0 in BackendRun () from postmaster
#26 0x0000000100298758 in BackendStartup () from postmaster
#27 0x0000000100297db0 in ServerLoop () from postmaster
#28 0x0000000100009b90 in PostmasterMain () from postmaster
#29 0x0000000100000680 in main () from postmaster
#30 0x000000010000028c in __start () from postmaster
(gdb) cont
Continuing.

Breakpoint 1, 0x00000001000019dc in errfinish () from postmaster
(gdb) bt
#0 0x00000001000019dc in errfinish () from postmaster
#1 0x0000000100002c58 in elog_finish () from postmaster
#2 0x0000000100007aa8 in proc_exit () from postmaster
#3 0x0000000100001c5c in errfinish () from postmaster
#4 0x000000010000a680 in AllocSetAlloc () from postmaster
#5 0x0000000100002a1c in MemoryContextAlloc () from postmaster
#6 0x0000000100108c28 in _bt_search () from postmaster
#7 0x0000000100106484 in _bt_first () from postmaster
#8 0x00000001001045b4 in btgettuple () from postmaster
#9 0x0000000100029fb0 in FunctionCall2 () from postmaster
#10 0x00000001000295f8 in index_getnext () from postmaster
#11 0x000000010002942c in systable_getnext () from postmaster
#12 0x000000010000f9a0 in ScanPgRelation () from postmaster
#13 0x0000000100011088 in RelationBuildDesc () from postmaster
#14 0x000000010000e6fc in RelationSysNameGetRelation () from postmaster
#15 0x000000010000e620 in relation_openr () from postmaster
#16 0x000000010000e44c in heap_openr () from postmaster
#17 0x0000000100041044 in RelationBuildTriggers () from postmaster
#18 0x00000001000111a4 in RelationBuildDesc () from postmaster
#19 0x000000010000e6fc in RelationSysNameGetRelation () from postmaster
#20 0x000000010000e620 in relation_openr () from postmaster
#21 0x000000010000e44c in heap_openr () from postmaster
#22 0x000000010000e1c8 in CatalogCacheInitializeCache () from postmaster
#23 0x000000010000dab8 in SearchCatCache () from postmaster
#24 0x000000010000da3c in SearchSysCache () from postmaster
#25 0x000000010028b570 in InitializeSessionUserId () from postmaster
#26 0x0000000100288ae8 in InitPostgres () from postmaster
#27 0x000000010029e2a8 in PostgresMain () from postmaster
#28 0x00000001002990e0 in BackendRun () from postmaster
#29 0x0000000100298758 in BackendStartup () from postmaster
#30 0x0000000100297db0 in ServerLoop () from postmaster
#31 0x0000000100009b90 in PostmasterMain () from postmaster
#32 0x0000000100000680 in main () from postmaster
#33 0x000000010000028c in __start () from postmaster
(gdb) cont
Continuing.

Breakpoint 1, 0x00000001000019dc in errfinish () from postmaster
(gdb) bt
#0 0x00000001000019dc in errfinish () from postmaster
#1 0x0000000100002c58 in elog_finish () from postmaster
#2 0x0000000100007bcc in shmem_exit () from postmaster
#3 0x0000000100007ab4 in proc_exit () from postmaster
#4 0x0000000100001c5c in errfinish () from postmaster
#5 0x000000010000a680 in AllocSetAlloc () from postmaster
#6 0x0000000100002a1c in MemoryContextAlloc () from postmaster
#7 0x0000000100108c28 in _bt_search () from postmaster
#8 0x0000000100106484 in _bt_first () from postmaster
#9 0x00000001001045b4 in btgettuple () from postmaster
#10 0x0000000100029fb0 in FunctionCall2 () from postmaster
#11 0x00000001000295f8 in index_getnext () from postmaster
#12 0x000000010002942c in systable_getnext () from postmaster
#13 0x000000010000f9a0 in ScanPgRelation () from postmaster
#14 0x0000000100011088 in RelationBuildDesc () from postmaster
#15 0x000000010000e6fc in RelationSysNameGetRelation () from postmaster
#16 0x000000010000e620 in relation_openr () from postmaster
#17 0x000000010000e44c in heap_openr () from postmaster
#18 0x0000000100041044 in RelationBuildTriggers () from postmaster
#19 0x00000001000111a4 in RelationBuildDesc () from postmaster
#20 0x000000010000e6fc in RelationSysNameGetRelation () from postmaster
#21 0x000000010000e620 in relation_openr () from postmaster
#22 0x000000010000e44c in heap_openr () from postmaster
#23 0x000000010000e1c8 in CatalogCacheInitializeCache () from postmaster
#24 0x000000010000dab8 in SearchCatCache () from postmaster
#25 0x000000010000da3c in SearchSysCache () from postmaster
#26 0x000000010028b570 in InitializeSessionUserId () from postmaster
#27 0x0000000100288ae8 in InitPostgres () from postmaster
#28 0x000000010029e2a8 in PostgresMain () from postmaster
#29 0x00000001002990e0 in BackendRun () from postmaster
#30 0x0000000100298758 in BackendStartup () from postmaster
#31 0x0000000100297db0 in ServerLoop () from postmaster
#32 0x0000000100009b90 in PostmasterMain () from postmaster
#33 0x0000000100000680 in main () from postmaster
#34 0x000000010000028c in __start () from postmaster
(gdb) cont
Continuing.

Breakpoint 1, 0x00000001000019dc in errfinish () from postmaster
(gdb) bt
#0 0x00000001000019dc in errfinish () from postmaster
#1 0x0000000100002c58 in elog_finish () from postmaster
#2 0x0000000100007b3c in proc_exit () from postmaster
#3 0x0000000100001c5c in errfinish () from postmaster
#4 0x000000010000a680 in AllocSetAlloc () from postmaster
#5 0x0000000100002a1c in MemoryContextAlloc () from postmaster
#6 0x0000000100108c28 in _bt_search () from postmaster
#7 0x0000000100106484 in _bt_first () from postmaster
#8 0x00000001001045b4 in btgettuple () from postmaster
#9 0x0000000100029fb0 in FunctionCall2 () from postmaster
#10 0x00000001000295f8 in index_getnext () from postmaster
#11 0x000000010002942c in systable_getnext () from postmaster
#12 0x000000010000f9a0 in ScanPgRelation () from postmaster
#13 0x0000000100011088 in RelationBuildDesc () from postmaster
#14 0x000000010000e6fc in RelationSysNameGetRelation () from postmaster
#15 0x000000010000e620 in relation_openr () from postmaster
#16 0x000000010000e44c in heap_openr () from postmaster
#17 0x0000000100041044 in RelationBuildTriggers () from postmaster
#18 0x00000001000111a4 in RelationBuildDesc () from postmaster
#19 0x000000010000e6fc in RelationSysNameGetRelation () from postmaster
#20 0x000000010000e620 in relation_openr () from postmaster
#21 0x000000010000e44c in heap_openr () from postmaster
#22 0x000000010000e1c8 in CatalogCacheInitializeCache () from postmaster
#23 0x000000010000dab8 in SearchCatCache () from postmaster
#24 0x000000010000da3c in SearchSysCache () from postmaster
#25 0x000000010028b570 in InitializeSessionUserId () from postmaster
#26 0x0000000100288ae8 in InitPostgres () from postmaster
#27 0x000000010029e2a8 in PostgresMain () from postmaster
#28 0x00000001002990e0 in BackendRun () from postmaster
#29 0x0000000100298758 in BackendStartup () from postmaster
#30 0x0000000100297db0 in ServerLoop () from postmaster
#31 0x0000000100009b90 in PostmasterMain () from postmaster
#32 0x0000000100000680 in main () from postmaster
#33 0x000000010000028c in __start () from postmaster
(gdb) cont
Continuing.

Program exited normally.
(gdb)

-----Original Message-----
From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: Tuesday, November 14, 2006 11:01 AM
To: Rob Owen
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: [GENERAL] Out of memory (Failed on request size 24)

"Rob Owen" <Rob(dot)Owen(at)sas(dot)com> writes:
> PostgreSQL 8.0.3 running on AIX 5.3 (same thing happens on 5.1 though).
> DBMS was running fine for some months but now one of the databases isn't accessible. Any help would be greatly appreciated.

Just one database? Sounds like it might be corrupt data in that database's system catalogs. Can you get a stack trace from the point of the error to help us narrow it down?

The way I usually debug startup-time failures is:

export PGOPTIONS="-W 30"
psql ...

Now I have 30 seconds to identify the PID of the backend process in another window and do (as the postgres user)

gdb /path/to/postgres PID

Once you've got gdb control of the backend, do

gdb> break errfinish
gdb> cont

... wait for the timeout to finish elapsing, if needed ... Once gdb reports that the breakpoint has been reached, say

gdb> bt
... useful info here...
gdb> cont

regards, tom lane