Creating foreign key on partitioned table is too slow

From: "kato-sho(at)fujitsu(dot)com" <kato-sho(at)fujitsu(dot)com>
To: "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Creating foreign key on partitioned table is too slow
Date: 2019-10-23 05:59:01
Message-ID: OSAPR01MB374809E8DE169C8BF2B82CBD9F6B0@OSAPR01MB3748.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello

To benchmark with tpcb model, I tried to create a foreign key in the partitioned history table, but backend process killed by OOM.
the number of partitions is 8192. I tried in master(commit: ad4b7aeb84).

I did the same thing in another server which has 200GB memory, but creating foreign key did not end in 24 hours.

Is the community aware of this? is anyone working on this?
If you are discussing, please let me know the thread.

Table definition and pstack are as follows.

* table definition *

CREATE TABLE accounts (aid INTEGER, bid INTEGER, abalance INTEGER, filler CHAR(84)) PARTITION BY HASH(aid);
CREATE TABLE history (tid INTEGER, bid INTEGER, aid INTEGER, delta INTEGER, mtime TIMESTAMP, filler CHAR(22)) PARTITION BY HASH(aid);
\o /dev/null
SELECT 'CREATE TABLE accounts_' || p || ' PARTITION OF accounts FOR VALUES WITH (modulus 8192, remainder ' || p || ');' FROM generate_series(0, 8191) p;
\gexec
SELECT 'CREATE TABLE history_' || p || ' PARTITION OF history FOR VALUES WITH (modulus 8192, remainder ' || p || ');' FROM generate_series(0, 8191) p;
\gexec
\o
ALTER TABLE accounts ADD CONSTRAINT accounts_pk PRIMARY KEY (aid);
ALTER TABLE history ADD CONSTRAINT history_fk3 FOREIGN KEY (aid) REFERENCES accounts (aid);

* pstack before killed by OOM *

#0 0x0000000000a84aec in ReleaseSysCache (tuple=0x7fbb0a15dc28) at syscache.c:1175
#1 0x0000000000a7135d in get_rel_relkind (relid=164628) at lsyscache.c:1816
#2 0x0000000000845f0a in RelationBuildPartitionDesc (rel=0x7fbadb9bfb10) at partdesc.c:230
#3 0x0000000000a78b9a in RelationBuildDesc (targetRelId=139268, insertIt=false) at relcache.c:1173
#4 0x0000000000a7b52e in RelationClearRelation (relation=0x7fbb0a1393e8, rebuild=true) at relcache.c:2534
#5 0x0000000000a7bacf in RelationFlushRelation (relation=0x7fbb0a1393e8) at relcache.c:2692
#6 0x0000000000a7bbe1 in RelationCacheInvalidateEntry (relationId=139268) at relcache.c:2744
#7 0x0000000000a6e11d in LocalExecuteInvalidationMessage (msg=0x7fbadb62e480) at inval.c:589
#8 0x0000000000a6de7d in ProcessInvalidationMessages (hdr=0x1d36d48, func=0xa6e01a <LocalExecuteInvalidationMessage>) at inval.c:460
#9 0x0000000000a6e94e in CommandEndInvalidationMessages () at inval.c:1095
#10 0x0000000000559c93 in AtCCI_LocalCache () at xact.c:1458
#11 0x00000000005596ac in CommandCounterIncrement () at xact.c:1040
#12 0x00000000006b1811 in addFkRecurseReferenced (wqueue=0x7fffcb0a0588, fkconstraint=0x20cf6a0, rel=0x7fbb0a1393e8, pkrel=0x7fbadb9bbe90, indexOid=189582, parentConstr=204810, numfks=1, pkattnum=0x7fffcb0a0190, fkattnum=0x7fffcb0a0150, pfeqoperators=0x7fffcb09ff50, ppeqoperators=0x7fffcb09fed0, ffeqoperators=0x7fffcb09fe50, old_check_ok=false) at tablecmds.c:8168
#13 0x00000000006b1a0b in addFkRecurseReferenced (wqueue=0x7fffcb0a0588, fkconstraint=0x20cf6a0, rel=0x7fbb0a1393e8, pkrel=0x7fbadc188840, indexOid=188424, parentConstr=0, numfks=1, pkattnum=0x7fffcb0a0190, fkattnum=0x7fffcb0a0150, pfeqoperators=0x7fffcb09ff50, ppeqoperators=0x7fffcb09fed0, ffeqoperators=0x7fffcb09fe50, old_check_ok=false) at tablecmds.c:8219
#14 0x00000000006b13e0 in ATAddForeignKeyConstraint (wqueue=0x7fffcb0a0588, tab=0x20cf4d8, rel=0x7fbb0a1393e8, fkconstraint=0x20cf6a0, parentConstr=0, recurse=true, recursing=false, lockmode=6) at tablecmds.c:8005
#15 0x00000000006afa0c in ATExecAddConstraint (wqueue=0x7fffcb0a0588, tab=0x20cf4d8, rel=0x7fbb0a1393e8, newConstraint=0x20cf6a0, recurse=true, is_readd=false, lockmode=6) at tablecmds.c:7419
#16 0x00000000006a8a7a in ATExecCmd (wqueue=0x7fffcb0a0588, tab=0x20cf4d8, rel=0x7fbb0a1393e8, cmd=0x20cf648, lockmode=6) at tablecmds.c:4300
#17 0x00000000006a8448 in ATRewriteCatalogs (wqueue=0x7fffcb0a0588, lockmode=6) at tablecmds.c:4185
#18 0x00000000006a7bf9 in ATController (parsetree=0x1cb4350, rel=0x7fbb0a1393e8, cmds=0x20cf428, recurse=true, lockmode=6) at tablecmds.c:3843
#19 0x00000000006a78a4 in AlterTable (relid=139268, lockmode=6, stmt=0x1cb4350) at tablecmds.c:3504
#20 0x0000000000914999 in ProcessUtilitySlow (pstate=0x1cb3a10, pstmt=0x1c91380, queryString=0x1c90170 "ALTER TABLE history ADD CONSTRAINT history_fk3 FOREIGN KEY (aid) REFERENCES accounts (aid);", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x1c91470, completionTag=0x7fffcb0a0d20 "") at utility.c:1131
#21 0x0000000000914490 in standard_ProcessUtility (pstmt=0x1c91380, queryString=0x1c90170 "ALTER TABLE history ADD CONSTRAINT history_fk3 FOREIGN KEY (aid) REFERENCES accounts (aid);", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x1c91470, completionTag=0x7fffcb0a0d20 "") at utility.c:927
#22 0x0000000000913534 in ProcessUtility (pstmt=0x1c91380, queryString=0x1c90170 "ALTER TABLE history ADD CONSTRAINT history_fk3 FOREIGN KEY (aid) REFERENCES accounts (aid);", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x1c91470, completionTag=0x7fffcb0a0d20 "") at utility.c:360
#23 0x000000000091245a in PortalRunUtility (portal=0x1cf5ee0, pstmt=0x1c91380, isTopLevel=true, setHoldSnapshot=false, dest=0x1c91470, completionTag=0x7fffcb0a0d20 "") at pquery.c:1175
#24 0x0000000000912671 in PortalRunMulti (portal=0x1cf5ee0, isTopLevel=true, setHoldSnapshot=false, dest=0x1c91470, altdest=0x1c91470, completionTag=0x7fffcb0a0d20 "") at pquery.c:1321
#25 0x0000000000911ba6 in PortalRun (portal=0x1cf5ee0, count=9223372036854775807, isTopLevel=true, run_once=true, dest=0x1c91470, altdest=0x1c91470, completionTag=0x7fffcb0a0d20 "") at pquery.c:796
#26 0x000000000090b9ad in exec_simple_query (query_string=0x1c90170 "ALTER TABLE history ADD CONSTRAINT history_fk3 FOREIGN KEY (aid) REFERENCES accounts (aid);") at postgres.c:1231
#27 0x000000000090fd13 in PostgresMain (argc=1, argv=0x1cb9fb8, dbname=0x1cb9ed0 "postgres", username=0x1cb9eb0 "k5user") at postgres.c:4256
#28 0x0000000000864cbd in BackendRun (port=0x1cb1e90) at postmaster.c:4498
#29 0x000000000086449b in BackendStartup (port=0x1cb1e90) at postmaster.c:4189
#30 0x00000000008608d7 in ServerLoop () at postmaster.c:1727
#31 0x000000000086018d in PostmasterMain (argc=1, argv=0x1c8aa40) at postmaster.c:1400
#32 0x0000000000770835 in main (argc=1, argv=0x1c8aa40) at main.c:210

regards,

Kato Sho

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Kyotaro Horiguchi 2019-10-23 06:04:10 Re: configure fails for perl check on CentOS8
Previous Message Peter Eisentraut 2019-10-23 05:13:20 Re: Clean up MinGW def file generation