Large objects and out-of-memory

From: Konstantin Knizhnik <k(dot)knizhnik(at)postgrespro(dot)ru>
To: pgsql-bugs(at)lists(dot)postgresql(dot)org
Subject: Large objects and out-of-memory
Date: 2020-12-21 16:33:42
Message-ID: 7c57860c-fc05-143e-aa81-f11ca13ec00a@postgrespro.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

Hi,

The following sequence of command cause backend's memory to exceed 10Gb:

CREATE DATABASE testmem;
CREATE ROLE alice WITH SUPERUSER LOGIN;
CREATE ROLE testlo WITH SUPERUSER LOGIN;
\c testmem # Подключаемся к тестовой базе
\c - alice
CREATE TABLE image1 (raster oid);
INSERT INTO image1 SELECT lo_creat(-1) FROM generate_series(1,10000000);
REASSIGN OWNED BY alice TO testlo;

This memory is occupied in top memory context by invalidation messages:

#0  AllocSetAlloc (context=0x19153a0, size=528) at aset.c:919
#1  0x0000000000b451d5 in MemoryContextAlloc (context=0x19153a0,
size=528) at mcxt.c:809
#2  0x0000000000ae80c6 in AddInvalidationMessage (listHdr=0x19156d8,
msg=0x7ffdca880600) at inval.c:241
#3  0x0000000000ae8473 in AddSnapshotInvalidationMessage (hdr=0x19156d0,
dbId=0, relId=1214) at inval.c:437
#4  0x0000000000ae871c in RegisterSnapshotInvalidation (dbId=0,
relId=1214) at inval.c:547
#5  0x0000000000ae915f in CacheInvalidateHeapTuple
(relation=0x7f16afc35cb0, tuple=0x7ffdca8807b0, newtuple=0x198dba8) at
inval.c:1162
#6  0x00000000004f2d41 in heap_update (relation=0x7f16afc35cb0,
otid=0x198dbac, newtup=0x198dba8, cid=6, crosscheck=0x0, wait=true,
tmfd=0x7ffdca880830,
    lockmode=0x7ffdca880828) at heapam.c:3729
#7  0x00000000004f3913 in simple_heap_update (relation=0x7f16afc35cb0,
otid=0x198dbac, tup=0x198dba8) at heapam.c:3891
#8  0x00000000005c281d in CatalogTupleUpdate (heapRel=0x7f16afc35cb0,
otid=0x198dbac, tup=0x198dba8) at indexing.c:309
#9  0x00000000005e7713 in shdepChangeDep (sdepRel=0x7f16afc35cb0,
classid=2613, objid=11950311, objsubid=0, refclassid=1260, refobjid=32848,
    deptype=SHARED_DEPENDENCY_OWNER) at pg_shdepend.c:269
#10 0x00000000005e784c in changeDependencyOnOwner (classId=2613,
objectId=11950311, newOwnerId=32848) at pg_shdepend.c:316
#11 0x0000000000669844 in AlterObjectOwner_internal (rel=0x7f16afc34c90,
objectId=11950311, new_ownerId=32848) at alter.c:1050
#12 0x00000000005e9c8e in shdepReassignOwned (roleids=0x19aa138,
newrole=32848) at pg_shdepend.c:1587
#13 0x0000000000718aa2 in ReassignOwnedObjects (stmt=0x18f1830) at
user.c:1425
#14 0x000000000097bf7d in standard_ProcessUtility (pstmt=0x18f1b80,
queryString=0x18f0cd0 "REASSIGN OWNED BY alice TO testlo;",
    context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x18f1c70, qc=0x7ffdca881180) at utility.c:890
#15 0x000000000097b508 in ProcessUtility (pstmt=0x18f1b80,
queryString=0x18f0cd0 "REASSIGN OWNED BY alice TO testlo;",
context=PROCESS_UTILITY_TOPLEVEL,

As far as I understand invalidation messages  are kept until the end of
transaction:

/*
 * CommandEndInvalidationMessages
 *        Process queued-up invalidation messages at end of one command
 *        in a transaction.
 *
 * Here, we send no messages to the shared queue, since we don't know
yet if
 * we will commit.  We do need to locally process the CurrentCmdInvalidMsgs
 * list, so as to flush our caches of any entries we have outdated in the
 * current command.  We then move the current-cmd list over to become part
 * of the prior-cmds list.
 *
 * Note:
 *        This should be called during CommandCounterIncrement(),
 *        after we have advanced the command ID.
 */

void
CommandEndInvalidationMessages(void)
{
AppendInvalidationMessages(&transInvalInfo->PriorCmdInvalidMsgs,
&transInvalInfo->CurrentCmdInvalidMsgs);
}

So in PriorCmdInvalidMsgs we have 10 millions invalidation message chunks.
Such memory blow happens in this scenario twice: when large objects are
created and when owner is reassigned.
It can not be considered as memory leak, but I think that it is a real
problem which has to be fixed.
I am not so familiar with invalidation message processing, I will be
pleae if somebody can suggest solution of the problem.

Thanks in advance,

--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message PG Bug reporting form 2020-12-21 17:00:01 BUG #16784: Server crash in ExecReScanAgg()
Previous Message PG Bug reporting form 2020-12-21 15:14:14 BUG #16783: pg_rewind: option -c does not work when configuration files are outside data directory