ERROR: out of memory, running insert/select

Here's another memory error I can reliably reproduce on postgres
8.1.3. This is on a different server, but with the same postgresql
config as my previous report. It was running a script to upgrade a
snapshot of a production database to the latest schema version for
testing. Eventually we will use this script to upgrade production
itself.

Here is the relevant output from the backend log:

TopMemoryContext: 119320 total in 13 blocks; 8416 free (23 chunks);
110904 used
SPI Plan: 3072 total in 2 blocks; 80 free (0 chunks); 2992 used
RI query cache: 24576 total in 2 blocks; 14136 free (5 chunks); 10440
used
Type information cache: 8192 total in 1 blocks; 1864 free (0 chunks);
6328 used
Operator class cache: 8192 total in 1 blocks; 4936 free (0 chunks);
3256 used
TopTransactionContext: -1276125184 total in 385 blocks; 16128 free
(402 chunks); -1276141312 used
MessageContext: 24576 total in 2 blocks; 1832 free (4 chunks); 22744
used
smgr relation table: 8192 total in 1 blocks; 808 free (0 chunks);
7384 used
Portal hash: 8192 total in 1 blocks; 3912 free (0 chunks); 4280 used
PortalMemory: 8192 total in 1 blocks; 8040 free (0 chunks); 152 used
PortalHeapMemory: 1024 total in 1 blocks; 896 free (0 chunks); 128 used
ExecutorState: 8912976 total in 12 blocks; 8363480 free (109247
chunks); 549496 used
ExprContext: 0 total in 0 blocks; 0 free (0 chunks); 0 used
ExprContext: 0 total in 0 blocks; 0 free (0 chunks); 0 used
ExprContext: 0 total in 0 blocks; 0 free (0 chunks); 0 used
ExprContext: 8192 total in 1 blocks; 8144 free (0 chunks); 48 used
Relcache by OID: 8192 total in 1 blocks; 1816 free (0 chunks); 6376 used
CacheMemoryContext: 1040384 total in 7 blocks; 437648 free (283
chunks); 602736 used
station_listener_id_initial_music_id: 1024 total in 1 blocks; 328
free (0 chunks); 696 used
listener_default_to_tracking_code_tracking_code_idx: 1024 total in 1
blocks; 392 free (0 chunks); 632 used
tracking_code_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632
used
listener_default_to_tracking_code_pkey: 1024 total in 1 blocks; 328
free (0 chunks); 696 used
pg_toast_27386_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
listener_default_webname_idx: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
listener_default_username_idx: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
listener_default_pkey: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
vendor_alias_v_alias_idx: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
vendor_alias_l_v_idx: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
vendor_alias_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
song_bookmark_station_id: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
song_bookmark_s_l_music_id_idx: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
song_bookmark_listener_id: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
song_bookmark_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632
used
pg_toast_27317_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
feedback_music_id_idx: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
feedback_date_created_is_positive_idx: 1024 total in 1 blocks; 328
free (0 chunks); 696 used
feedback_station_id_idx: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
feedback_station_music_id_idx: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
feedback_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_toast_27268_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
seed_music_id_idx: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
seed_station_id: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
seed_station_music_id_idx: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
seed_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
station_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
email_listener_id_idx: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
email_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_toast_27162_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
settlement_log_credit_card_id_idx: 1024 total in 1 blocks; 392 free
(0 chunks); 632 used
settlement_log_listener_id_idx: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
settlement_log_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632
used
pg_toast_27130_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
payment_listener_id_idx: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
payment_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_toast_27090_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
credit_card_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_toast_27066_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
tired_song_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
tired_song_listener_song_identity_idx: 1024 total in 1 blocks; 328
free (0 chunks); 696 used
pg_toast_27032_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
listener_login_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632
used
pg_toast_27007_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
listener_to_station_station_id_idx: 1024 total in 1 blocks; 392 free
(0 chunks); 632 used
listener_to_station_pkey: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
listener_expiration_date_idx: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
listener_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_toast_26983_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
pg_description_o_c_o_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
legacy_station_mapping_pkey: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_trigger_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_trigger_tgconstrrelid_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_trigger_tgconstrname_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_toast_26915_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
pg_attrdef_adrelid_adnum_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_attrdef_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
cluster_segment_uniqueness_idx: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
cluster_segment_pkey: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_constraint_contypid_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_constraint_conrelid_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_constraint_conname_nsp_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_constraint_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_toast_26907_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
pg_shdepend_depender_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_shdepend_reference_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_depend_depender_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_depend_reference_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
radio_info_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_index_indrelid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_type_typname_nsp_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_type_oid_index: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_trigger_tgrelid_tgname_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_statistic_relid_att_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_auth_members_member_role_index: 1024 total in 1 blocks; 328 free
(0 chunks); 696 used
pg_auth_members_role_member_index: 1024 total in 1 blocks; 328 free
(0 chunks); 696 used
pg_rewrite_rel_rulename_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_proc_proname_args_nsp_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_proc_oid_index: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_operator_oprname_l_r_n_index: 1024 total in 1 blocks; 192 free (0
chunks); 832 used
pg_operator_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_opclass_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_opclass_am_name_nsp_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_namespace_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_namespace_nspname_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_language_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_language_name_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_inherits_relid_seqno_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_index_indexrelid_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_authid_oid_index: 1024 total in 1 blocks; 392 free (0 chunks); 632
used
pg_authid_rolname_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_database_datname_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_conversion_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_toast_26907_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
pg_shdepend_depender_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_shdepend_reference_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_depend_depender_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_depend_reference_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
radio_info_pkey: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_index_indrelid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_type_typname_nsp_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_type_oid_index: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_trigger_tgrelid_tgname_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_statistic_relid_att_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_auth_members_member_role_index: 1024 total in 1 blocks; 328 free
(0 chunks); 696 used
pg_auth_members_role_member_index: 1024 total in 1 blocks; 328 free
(0 chunks); 696 used
pg_rewrite_rel_rulename_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_proc_proname_args_nsp_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_proc_oid_index: 1024 total in 1 blocks; 392 free (0 chunks); 632 used
pg_operator_oprname_l_r_n_index: 1024 total in 1 blocks; 192 free (0
chunks); 832 used
pg_operator_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_opclass_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_opclass_am_name_nsp_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_namespace_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_namespace_nspname_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_language_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_language_name_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_inherits_relid_seqno_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_index_indexrelid_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_authid_oid_index: 1024 total in 1 blocks; 392 free (0 chunks); 632
used
pg_authid_rolname_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_database_datname_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
pg_conversion_oid_index: 1024 total in 1 blocks; 392 free (0 chunks);
632 used
pg_conversion_name_nsp_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_conversion_default_index: 1024 total in 1 blocks; 192 free (0
chunks); 832 used
pg_class_relname_nsp_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_class_oid_index: 1024 total in 1 blocks; 392 free (0 chunks); 632
used
pg_cast_source_target_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_attribute_relid_attnum_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_attribute_relid_attnam_index: 1024 total in 1 blocks; 328 free (0
chunks); 696 used
pg_amproc_opc_proc_index: 1024 total in 1 blocks; 256 free (0
chunks); 768 used
pg_amop_opr_opc_index: 1024 total in 1 blocks; 328 free (0 chunks);
696 used
pg_amop_opc_strat_index: 1024 total in 1 blocks; 256 free (0 chunks);
768 used
pg_aggregate_fnoid_index: 1024 total in 1 blocks; 392 free (0
chunks); 632 used
MdSmgr: 8192 total in 1 blocks; 5056 free (5 chunks); 3136 used
LockTable (locallock hash): 57344 total in 3 blocks; 33496 free (8
chunks); 23848 used
Timezones: 47592 total in 2 blocks; 5968 free (0 chunks); 41624 used
ErrorContext: 8192 total in 1 blocks; 8176 free (5 chunks); 16 used
2006-05-04 10:08:48.025 PDT [d:radio_source u:slony s:4459986e.570d
75] ERROR: out of memory
2006-05-04 10:08:48.025 PDT [d:radio_source u:slony s:4459986e.570d
76] DETAIL: Failed on request of size 32.
2006-05-04 10:08:48.025 PDT [d:radio_source u:slony s:4459986e.570d
77] STATEMENT: INSERT INTO station
(station_id, name, listener_id, shared_station_id,
time_added,
shared_creator_id)
SELECT nextval('station_id_seq'), s.name, lts.listener_id,
s.station_id,
lts.time_added, s.listener_id
FROM station AS s, listener_to_station AS lts
WHERE lts.listener_id != s.listener_id;

After this statement fails, a number of others are executed, all
resulting in memory errors. Eventually the backend fails altogether
with:

2006-05-04 10:08:48.062 PDT [d:radio_source u:slony s:4459986e.570d
87] ERROR: out of memory
2006-05-04 10:08:48.062 PDT [d:radio_source u:slony s:4459986e.570d
88] DETAIL: Failed on request of size 2112.
2006-05-04 10:08:48.062 PDT [d:radio_source u:slony s:4459986e.570d
89] PANIC: ERRORDATA_STACK_SIZE exceeded
2006-05-04 10:08:48.653 PDT LOG: server process (PID 22285) was
terminated by signal 6
2006-05-04 10:08:48.653 PDT LOG: terminating any other active server
processes

There is much intervening output before this with similar error
output as the first query (but the statements themselves are not
logged). If you need it, I can include that output as well.

The original failing statement is populating a new table from an
upgraded one. I can't really provide explain output easily because
the schema was significantly modified by other statements in the
upgrade script before getting to the failing update. The upgrade
script takes many hours to run, so I figured I would submit what I
know now. If you need other output, let me know and I will try and
extract it.

Thanks.

-Casey