Re: Out of memory (Failed on request size 24)

Thanks Martijn,
I reduced a number of the buffers and connection settings, and added some more tracing and this is the result. The number (TopTransactionContext) is smaller, but still very large. Any reason why this number would suddenly go sky high - the same system was working fine just a month ago.

<2006-11-14 05:48:35 EST>LOG: 00000: database system was shut down at 2006-11-14 05:48:30 EST
<2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4049
<2006-11-14 05:48:35 EST>LOG: 00000: checkpoint record is at 0/19A825B8
<2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4132
<2006-11-14 05:48:35 EST>LOG: 00000: redo record is at 0/19A825B8; undo record is at 0/0; shutdown TRUE
<2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4160
<2006-11-14 05:48:35 EST>LOG: 00000: next transaction ID: 626916; next OID: 54355
<2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4163
<2006-11-14 05:48:35 EST>LOG: 00000: database system is ready
<2006-11-14 05:48:35 EST>LOCATION: StartupXLOG, xlog.c:4526
<2006-11-14 05:48:35 EST>DEBUG: 00000: proc_exit(0)
<2006-11-14 05:48:35 EST>LOCATION: proc_exit, ipc.c:95
<2006-11-14 05:48:35 EST>DEBUG: 00000: shmem_exit(0)
<2006-11-14 05:48:35 EST>LOCATION: shmem_exit, ipc.c:126
<2006-11-14 05:48:35 EST>DEBUG: 00000: exit(0)
<2006-11-14 05:48:35 EST>LOCATION: proc_exit, ipc.c:113
<2006-11-14 05:48:35 EST>DEBUG: 00000: reaping dead processes
<2006-11-14 05:48:35 EST>LOCATION: reaper, postmaster.c:1988
<2006-11-14 05:48:46 EST>DEBUG: 00000: forked new backend, pid=168246 socket=7
<2006-11-14 05:48:46 EST>LOCATION: BackendStartup, postmaster.c:2499
<2006-11-14 05:48:46 EST>DEBUG: 00000: postmaster child[168246]: starting with (
<2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2829
<2006-11-14 05:48:46 EST>DEBUG: 00000: postgres
<2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2832
<2006-11-14 05:48:46 EST>DEBUG: 00000: -v196608
<2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2832
<2006-11-14 05:48:46 EST>DEBUG: 00000: -p
<2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2832
<2006-11-14 05:48:46 EST>DEBUG: 00000: files
<2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2832
<2006-11-14 05:48:46 EST>DEBUG: 00000: )
<2006-11-14 05:48:46 EST>LOCATION: BackendRun, postmaster.c:2834
<2006-11-14 05:48:46 EST>DEBUG: 00000: InitPostgres
<2006-11-14 05:48:46 EST>LOCATION: PostgresMain, postgres.c:2719
<2006-11-14 05:48:46 EST>DEBUG: 00000: StartTransaction
<2006-11-14 05:48:46 EST>LOCATION: ShowTransactionState, xact.c:3609
<2006-11-14 05:48:46 EST>DEBUG: 00000: name: unnamed; blockState: DEFAULT; state: INPROGR, xid/subid/cid: 626916/
1/0, nestlvl: 1, children: <>
<2006-11-14 05:48:46 EST>LOCATION: ShowTransactionStateRec, xact.c:3634
TopMemoryContext: 32768 total in 3 blocks; 10760 free (3 chunks); 22008 used
TopTransactionContext: 1340071936 total in 170 blocks; 928 free (14 chunks); 1340071008 used
PortalMemory: 0 total in 0 blocks; 0 free (0 chunks); 0 used
CacheMemoryContext: 516096 total in 6 blocks; 178752 free (10 chunks); 337344 used
pg_operator_oid_index: 1024 total in 1 blocks; 840 free (0 chunks); 184 used
pg_amproc_opc_proc_index: 1024 total in 1 blocks; 736 free (0 chunks); 288 used
pg_amop_opc_strat_index: 1024 total in 1 blocks; 736 free (0 chunks); 288 used
pg_index_indexrelid_index: 1024 total in 1 blocks; 840 free (0 chunks); 184 used
pg_attribute_relid_attnum_index: 1024 total in 1 blocks; 744 free (0 chunks); 280 used
pg_class_oid_index: 1024 total in 1 blocks; 840 free (0 chunks); 184 used
pg_amproc_opc_proc_index: 1024 total in 1 blocks; 736 free (0 chunks); 288 used
pg_amop_opc_strat_index: 1024 total in 1 blocks; 736 free (0 chunks); 288 used
pg_class_relname_nsp_index: 1024 total in 1 blocks; 744 free (0 chunks); 280 used
MdSmgr: 8192 total in 1 blocks; 7808 free (0 chunks); 384 used
DynaHash: 8192 total in 1 blocks; 5936 free (0 chunks); 2256 used
Operator class cache: 8192 total in 1 blocks; 1968 free (0 chunks); 6224 used
smgr relation table: 24576 total in 2 blocks; 16256 free (5 chunks); 8320 used
Portal hash: 8192 total in 1 blocks; 4032 free (0 chunks); 4160 used
Relcache by OID: 8192 total in 1 blocks; 928 free (0 chunks); 7264 used
Relcache by name: 24576 total in 2 blocks; 14208 free (5 chunks); 10368 used
LockTable (locallock hash): 24576 total in 2 blocks; 16272 free (6 chunks); 8304 used
ErrorContext: 8192 total in 1 blocks; 8160 free (7 chunks); 32 used
<2006-11-14 05:50:03 EST>FATAL: 53200: out of memory
<2006-11-14 05:50:03 EST>DETAIL: Failed on request of size 24.
<2006-11-14 05:50:03 EST>LOCATION: AllocSetAlloc, aset.c:702
<2006-11-14 05:50:03 EST>DEBUG: 00000: proc_exit(0)
<2006-11-14 05:50:03 EST>LOCATION: proc_exit, ipc.c:95
<2006-11-14 05:50:03 EST>DEBUG: 00000: shmem_exit(0)
<2006-11-14 05:50:03 EST>LOCATION: shmem_exit, ipc.c:126
<2006-11-14 05:50:03 EST>DEBUG: 00000: exit(0)
<2006-11-14 05:50:03 EST>LOCATION: proc_exit, ipc.c:113
<2006-11-14 05:50:03 EST>DEBUG: 00000: reaping dead processes
<2006-11-14 05:50:03 EST>LOCATION: reaper, postmaster.c:1988
<2006-11-14 05:50:03 EST>DEBUG: 00000: server process (PID 168246) exited with exit code 0
<2006-11-14 05:50:03 EST>LOCATION: LogChildExit, postmaster.c:2349
<2006-11-14 05:53:49 EST>DEBUG: 00000: postmaster received signal 15
<2006-11-14 05:53:49 EST>LOCATION: pmdie, postmaster.c:1850
<2006-11-14 05:53:49 EST>LOG: 00000: received smart shutdown request
<2006-11-14 05:53:49 EST>LOCATION: pmdie, postmaster.c:1865
<2006-11-14 05:53:49 EST>LOG: 00000: shutting down
<2006-11-14 05:53:49 EST>LOCATION: ShutdownXLOG, xlog.c:4706
<2006-11-14 05:53:49 EST>DEBUG: 00000: reaping dead processes
<2006-11-14 05:53:49 EST>LOCATION: reaper, postmaster.c:1988
<2006-11-14 05:53:49 EST>LOG: 00000: database system is shut down
<2006-11-14 05:53:49 EST>LOCATION: ShutdownXLOG, xlog.c:4715
<2006-11-14 05:53:49 EST>DEBUG: 00000: proc_exit(0)
<2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:95
<2006-11-14 05:53:49 EST>DEBUG: 00000: shmem_exit(0)
<2006-11-14 05:53:49 EST>LOCATION: shmem_exit, ipc.c:126
<2006-11-14 05:53:49 EST>DEBUG: 00000: exit(0)
<2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:113
<2006-11-14 05:53:49 EST>DEBUG: 00000: reaping dead processes
<2006-11-14 05:53:49 EST>LOCATION: reaper, postmaster.c:1988
<2006-11-14 05:53:49 EST>DEBUG: 00000: proc_exit(0)
<2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:95
<2006-11-14 05:53:49 EST>DEBUG: 00000: shmem_exit(0)
<2006-11-14 05:53:49 EST>LOCATION: shmem_exit, ipc.c:126
<2006-11-14 05:53:49 EST>DEBUG: 00000: exit(0)
<2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:113
<2006-11-14 05:53:49 EST>LOG: 00000: logger shutting down
<2006-11-14 05:53:49 EST>LOCATION: SysLoggerMain, syslogger.c:361
<2006-11-14 05:53:49 EST>DEBUG: 00000: proc_exit(0)
<2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:95
<2006-11-14 05:53:49 EST>DEBUG: 00000: shmem_exit(0)
<2006-11-14 05:53:49 EST>LOCATION: shmem_exit, ipc.c:126
<2006-11-14 05:53:49 EST>DEBUG: 00000: exit(0)
<2006-11-14 05:53:49 EST>LOCATION: proc_exit, ipc.c:113

-----Original Message-----
From: Martijn van Oosterhout [mailto:kleptog(at)svana(dot)org]
Sent: Tuesday, November 14, 2006 6:44 AM
To: Rob Owen
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: [GENERAL] Out of memory (Failed on request size 24)

On Tue, Nov 14, 2006 at 05:53:08AM -0500, Rob Owen wrote:
> PostgreSQL 8.0.3 running on AIX 5.3 (same thing happens on 5.1 though).
> DBMS was running fine for some months but now one of the databases isn't accessible. Any help would be greatly appreciated.
>
> DBMS starts up fine, but any operation on the files database (psql
> files, vaccumdb files, pgdump files) yields the same result. The
> client responds with
>
> > psql files

<snip>

Something screwed up:

> TopTransactionContext: 2145378304 total in 266 blocks; 928 free (14
> chunks); 2145377376 used

That's a lot of memory. I thought there was a check on negative sized allocations... Did "make check" pass ok?

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.