Re: BUG #16260: Repetitive accessing to partitioned table inside transaction causes server process crash

PG Bug reporting form <noreply(at)postgresql(dot)org> writes:
> Accessing to a partitioned table with large partitions (20K) inside a
> transaction can cause postgres to use large amount of memory (>30GB) and
> ultimately leading to server process crash.

This isn't particularly surprising.

> 2020-02-16 17:01:35.449 CST [22652] LOG: server process (PID 23149) was
> terminated by signal 9: Killed

Notice that this is an external SIGKILL, not an internal-to-postgres
crash. Presumably, the process got killed by Linux's infamous
OOM killer. We typically recommend that Postgres servers be set up
to avoid OOM kills by disabling kernel memory overcommit [1]. However,
the fundamental problem here is that lots of partitions translates to
lots of memory usage. Our current recommendation is to not exceed
"a few thousand" partitions [2].

There has been considerable work already on reducing the memory
consumption for heavily-partitioned queries, and we'll continue to work
on it, but don't hold your breath --- and definitely don't expect that
there will be any back-patched bug fixes for it. Significant forward
progress will probably require major rewrites, on the scale of what's
discussed in [3].

In short: use fewer partitions.

regards, tom lane

[1] https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT
[2] https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE-BEST-PRACTICES
[3] https://www.postgresql.org/message-id/flat/357.1550612935%40sss.pgh.pa.us