Re: Out of Memory errors are frustrating as heck!

On 4/15/2019 21:49, Gunther wrote:
>
> I'm going to try without that DISTINCT step, or perhaps by dismantling
> this query until it works without this excessive memory growth.
>
It also failed. Out of memory. The resident memory size of the backend
was 1.5 GB before it crashed.

TopMemoryContext: 4335600 total in 8 blocks; 41208 free (16 chunks);
4294392 used HandleParallelMessages: 8192 total in 1 blocks; 7936 free
(0 chunks); 256 used TableSpace cache: 8192 total in 1 blocks; 2096 free
(0 chunks); 6096 used Type information cache: 24352 total in 2 blocks;
2624 free (0 chunks); 21728 used Operator lookup cache: 24576 total in 2
blocks; 10760 free (3 chunks); 13816 used pgstat TabStatusArray lookup
hash table: 8192 total in 1 blocks; 416 free (0 chunks); 7776 used
TopTransactionContext: 8192 total in 1 blocks; 5416 free (2 chunks);
2776 used RowDescriptionContext: 8192 total in 1 blocks; 6896 free (0
chunks); 1296 used MessageContext: 524288 total in 7 blocks; 186848 free
(7 chunks); 337440 used Operator class cache: 8192 total in 1 blocks;
560 free (0 chunks); 7632 used smgr relation table: 32768 total in 3
blocks; 16832 free (8 chunks); 15936 used TransactionAbortContext: 32768
total in 1 blocks; 32512 free (0 chunks); 256 used Portal hash: 8192
total in 1 blocks; 560 free (0 chunks); 7632 used TopPortalContext: 8192
total in 1 blocks; 7664 free (0 chunks); 528 used PortalHoldContext:
24632 total in 2 blocks; 7392 free (0 chunks); 17240 used PortalContext:
1105920 total in 138 blocks; 10368 free (8 chunks); 1095552 used:
ExecutorState: 2238648944 total in 266772 blocks; 3726944 free (16276
chunks); 2234922000 used HashTableContext: 16384 total in 2 blocks; 4032
free (5 chunks); 12352 used HashBatchContext: 8192 total in 1 blocks;
7936 free (0 chunks); 256 used HashTableContext: 8192 total in 1 blocks;
7320 free (0 chunks); 872 used HashBatchContext: 8192 total in 1 blocks;
7936 free (0 chunks); 256 used HashTableContext: 8192 total in 1 blocks;
7320 free (0 chunks); 872 used HashBatchContext: 8192 total in 1 blocks;
7936 free (0 chunks); 256 used HashTableContext: 8192 total in 1 blocks;
7752 free (0 chunks); 440 used HashBatchContext: 90288 total in 4
blocks; 16072 free (6 chunks); 74216 used HashTableContext: 8192 total
in 1 blocks; 7624 free (0 chunks); 568 used HashBatchContext: 90288
total in 4 blocks; 16072 free (6 chunks); 74216 used TupleSort main:
286912 total in 8 blocks; 246792 free (39 chunks); 40120 used TupleSort
main: 286912 total in 8 blocks; 246792 free (39 chunks); 40120 used
ExprContext: 8192 total in 1 blocks; 7936 free (0 chunks); 256 used
HashTableContext: 8454256 total in 6 blocks; 64848 free (32 chunks);
8389408 used HashBatchContext: 66935744 total in 2037 blocks; 7936 free
(0 chunks); 66927808 used ExprContext: 8192 total in 1 blocks; 7936 free
(0 chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used ExprContext: 8192 total in 1 blocks; 7936 free (0
chunks); 256 used Relcache by OID: 16384 total in 2 blocks; 3512 free (2
chunks); 12872 used CacheMemoryContext: 1101328 total in 14 blocks;
383480 free (0 chunks); 717848 used index info: 2048 total in 2 blocks;
680 free (1 chunks); 1368 used: pg_toast_2619_index index info: 2048
total in 2 blocks; 968 free (1 chunks); 1080 used: entity_id_fkidx index
info: 2048 total in 2 blocks; 696 free (1 chunks); 1352 used:
entity_id_idx index info: 2048 total in 2 blocks; 968 free (1 chunks);
1080 used: act_id_fkidx index info: 2048 total in 2 blocks; 696 free (1
chunks); 1352 used: act_id_idx index info: 2048 total in 2 blocks; 592
free (1 chunks); 1456 used:
pg_constraint_conrelid_contypid_conname_index index info: 2048 total in
2 blocks; 952 free (1 chunks); 1096 used: actrelationship_pkey index
info: 2048 total in 2 blocks; 624 free (1 chunks); 1424 used:
actrelationship_target_idx index info: 2048 total in 2 blocks; 624 free
(1 chunks); 1424 used: actrelationship_source_idx index info: 2048 total
in 2 blocks; 680 free (1 chunks); 1368 used: documentinformation_pk
index info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_statistic_ext_relid_index index info: 2048 total in 2 blocks; 952
free (1 chunks); 1096 used: docinfsubj_ndx_seii index info: 2048 total
in 2 blocks; 952 free (1 chunks); 1096 used:
docinfsubj_ndx_sbjentcodeonly index info: 2048 total in 2 blocks; 952
free (1 chunks); 1096 used: pg_index_indrelid_index index info: 2048
total in 2 blocks; 648 free (2 chunks); 1400 used:
pg_db_role_setting_databaseid_rol_index index info: 2048 total in 2
blocks; 624 free (2 chunks); 1424 used: pg_opclass_am_name_nsp_index
index info: 1024 total in 1 blocks; 16 free (0 chunks); 1008 used:
pg_foreign_data_wrapper_name_index index info: 1024 total in 1 blocks;
48 free (0 chunks); 976 used: pg_enum_oid_index index info: 2048 total
in 2 blocks; 680 free (2 chunks); 1368 used: pg_class_relname_nsp_index
index info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_foreign_server_oid_index index info: 1024 total in 1 blocks; 48 free
(0 chunks); 976 used: pg_publication_pubname_index index info: 2048
total in 2 blocks; 592 free (3 chunks); 1456 used:
pg_statistic_relid_att_inh_index index info: 2048 total in 2 blocks; 680
free (2 chunks); 1368 used: pg_cast_source_target_index index info: 1024
total in 1 blocks; 48 free (0 chunks); 976 used: pg_language_name_index
index info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_transform_oid_index index info: 1024 total in 1 blocks; 48 free (0
chunks); 976 used: pg_collation_oid_index index info: 3072 total in 2
blocks; 1136 free (2 chunks); 1936 used: pg_amop_fam_strat_index index
info: 2048 total in 2 blocks; 952 free (1 chunks); 1096 used:
pg_index_indexrelid_index index info: 2048 total in 2 blocks; 760 free
(2 chunks); 1288 used: pg_ts_template_tmplname_index index info: 2048
total in 2 blocks; 704 free (3 chunks); 1344 used:
pg_ts_config_map_index index info: 2048 total in 2 blocks; 952 free (1
chunks); 1096 used: pg_opclass_oid_index index info: 1024 total in 1
blocks; 16 free (0 chunks); 1008 used: pg_foreign_data_wrapper_oid_index
index info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_event_trigger_evtname_index index info: 2048 total in 2 blocks; 760
free (2 chunks); 1288 used: pg_statistic_ext_name_index index info: 1024
total in 1 blocks; 48 free (0 chunks); 976 used:
pg_publication_oid_index index info: 1024 total in 1 blocks; 48 free (0
chunks); 976 used: pg_ts_dict_oid_index index info: 1024 total in 1
blocks; 48 free (0 chunks); 976 used: pg_event_trigger_oid_index index
info: 3072 total in 2 blocks; 1216 free (3 chunks); 1856 used:
pg_conversion_default_index index info: 3072 total in 2 blocks; 1136
free (2 chunks); 1936 used: pg_operator_oprname_l_r_n_index index info:
2048 total in 2 blocks; 680 free (2 chunks); 1368 used:
pg_trigger_tgrelid_tgname_index index info: 2048 total in 2 blocks; 760
free (2 chunks); 1288 used: pg_enum_typid_label_index index info: 1024
total in 1 blocks; 48 free (0 chunks); 976 used: pg_ts_config_oid_index
index info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_user_mapping_oid_index index info: 2048 total in 2 blocks; 704 free
(3 chunks); 1344 used: pg_opfamily_am_name_nsp_index index info: 1024
total in 1 blocks; 48 free (0 chunks); 976 used:
pg_foreign_table_relid_index index info: 2048 total in 2 blocks; 952
free (1 chunks); 1096 used: pg_type_oid_index index info: 1024 total in
1 blocks; 48 free (0 chunks); 976 used: pg_aggregate_fnoid_index index
info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_constraint_oid_index index info: 2048 total in 2 blocks; 760 free (2
chunks); 1288 used: pg_rewrite_rel_rulename_index index info: 2048 total
in 2 blocks; 760 free (2 chunks); 1288 used: pg_ts_parser_prsname_index
index info: 2048 total in 2 blocks; 760 free (2 chunks); 1288 used:
pg_ts_config_cfgname_index index info: 1024 total in 1 blocks; 48 free
(0 chunks); 976 used: pg_ts_parser_oid_index index info: 2048 total in 2
blocks; 728 free (1 chunks); 1320 used:
pg_publication_rel_prrelid_prpubid_index index info: 2048 total in 2
blocks; 952 free (1 chunks); 1096 used: pg_operator_oid_index index
info: 2048 total in 2 blocks; 952 free (1 chunks); 1096 used:
pg_namespace_nspname_index index info: 1024 total in 1 blocks; 48 free
(0 chunks); 976 used: pg_ts_template_oid_index index info: 2048 total in
2 blocks; 624 free (2 chunks); 1424 used: pg_amop_opr_fam_index index
info: 2048 total in 2 blocks; 672 free (3 chunks); 1376 used:
pg_default_acl_role_nsp_obj_index index info: 2048 total in 2 blocks;
704 free (3 chunks); 1344 used: pg_collation_name_enc_nsp_index index
info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_publication_rel_oid_index index info: 2048 total in 2 blocks; 952
free (1 chunks); 1096 used: pg_range_rngtypid_index index info: 2048
total in 2 blocks; 760 free (2 chunks); 1288 used:
pg_ts_dict_dictname_index index info: 2048 total in 2 blocks; 760 free
(2 chunks); 1288 used: pg_type_typname_nsp_index index info: 1024 total
in 1 blocks; 48 free (0 chunks); 976 used: pg_opfamily_oid_index index
info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_statistic_ext_oid_index index info: 2048 total in 2 blocks; 952 free
(1 chunks); 1096 used: pg_class_oid_index index info: 2048 total in 2
blocks; 704 free (3 chunks); 1344 used: pg_proc_proname_args_nsp_index
index info: 1024 total in 1 blocks; 16 free (0 chunks); 1008 used:
pg_partitioned_table_partrelid_index index info: 2048 total in 2 blocks;
760 free (2 chunks); 1288 used: pg_transform_type_lang_index index info:
2048 total in 2 blocks; 680 free (2 chunks); 1368 used:
pg_attribute_relid_attnum_index index info: 2048 total in 2 blocks; 952
free (1 chunks); 1096 used: pg_proc_oid_index index info: 1024 total in
1 blocks; 48 free (0 chunks); 976 used: pg_language_oid_index index
info: 1024 total in 1 blocks; 48 free (0 chunks); 976 used:
pg_namespace_oid_index index info: 3072 total in 2 blocks; 1136 free (2
chunks); 1936 used: pg_amproc_fam_proc_index index info: 1024 total in 1
blocks; 48 free (0 chunks); 976 used: pg_foreign_server_name_index index
info: 2048 total in 2 blocks; 760 free (2 chunks); 1288 used:
pg_attribute_relid_attnam_index index info: 1024 total in 1 blocks; 48
free (0 chunks); 976 used: pg_conversion_oid_index index info: 2048
total in 2 blocks; 728 free (1 chunks); 1320 used:
pg_user_mapping_user_server_index index info: 2048 total in 2 blocks;
728 free (1 chunks); 1320 used:
pg_subscription_rel_srrelid_srsubid_index index info: 1024 total in 1
blocks; 48 free (0 chunks); 976 used: pg_sequence_seqrelid_index index
info: 2048 total in 2 blocks; 760 free (2 chunks); 1288 used:
pg_conversion_name_nsp_index index info: 2048 total in 2 blocks; 952
free (1 chunks); 1096 used: pg_authid_oid_index index info: 2048 total
in 2 blocks; 728 free (1 chunks); 1320 used:
pg_auth_members_member_role_index index info: 1024 total in 1 blocks; 48
free (0 chunks); 976 used: pg_subscription_oid_index index info: 2048
total in 2 blocks; 952 free (1 chunks); 1096 used:
pg_tablespace_oid_index index info: 2048 total in 2 blocks; 704 free (3
chunks); 1344 used: pg_shseclabel_object_index index info: 1024 total in
1 blocks; 16 free (0 chunks); 1008 used:
pg_replication_origin_roname_index index info: 2048 total in 2 blocks;
952 free (1 chunks); 1096 used: pg_database_datname_index index info:
2048 total in 2 blocks; 760 free (2 chunks); 1288 used:
pg_subscription_subname_index index info: 1024 total in 1 blocks; 16
free (0 chunks); 1008 used: pg_replication_origin_roiident_index index
info: 2048 total in 2 blocks; 728 free (1 chunks); 1320 used:
pg_auth_members_role_member_index index info: 2048 total in 2 blocks;
952 free (1 chunks); 1096 used: pg_database_oid_index index info: 2048
total in 2 blocks; 952 free (1 chunks); 1096 used:
pg_authid_rolname_index WAL record construction: 49768 total in 2
blocks; 6368 free (0 chunks); 43400 used PrivateRefCount: 8192 total in
1 blocks; 2624 free (0 chunks); 5568 used MdSmgr: 8192 total in 1
blocks; 7208 free (1 chunks); 984 used LOCALLOCK hash: 16384 total in 2
blocks; 4600 free (2 chunks); 11784 used Timezones: 104120 total in 2
blocks; 2624 free (0 chunks); 101496 used ErrorContext: 8192 total in 1
blocks; 7936 free (5 chunks); 256 used Grand total: 2322733048 bytes in
269225 blocks; 5406896 free (16556 chunks); 2317326152 used

So what I am wondering now, is there seems to be an EXPLOSION of memory
consumption near the time of the crash. That ExecutorState has
2,238,648,944 but just until the very last second(s) the RES memory as
per top was 1.5 GB I swear. I looked at it. It went like this:

1.5 GB for a very long time
1.1 GB -- and I thought, yeah! it worked! it's shrinking now
and then it was gone, and there was the memory error.

So how can top tell me 1.5 GB while here the ExecutorState allocations
alone have 2 GB???

And isn't even 1.5 GB way too big?

Is there a way of dumping that memory map info during normal runtime, by
calling a function with the debugger? So one can see how it grows? It's
like checking out memory leaks with Java where I keep looking at the
heap_info summary.  Tom Lane said that this ExecutorState should not
grow to anything like this size, right?

-Gunther