Re: BUG #16925: ERROR: invalid DSA memory alloc request size 1073741824 CONTEXT: parallel worker

On Thu, Mar 18, 2021 at 2:06 AM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
> On Sat, Mar 13, 2021 at 11:59 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
> > > > 2021-03-12 19:45:37.352 CET [316243] ERROR: XX000: invalid DSA memory alloc
> > > > request size 1073741824
>
> After an off-list exchange with Karen and colleague who ran this with
> the ERROR changed to a PANIC and examined the smoldering core, the
> problem turns out to be a failure to keep the hashtable bucket array
> <= MaxAllocSize in one code path. Although commit 86a2218e fixed
> another version of that problem a while ago, it can still be
> exceeded... by one byte... when we expand from one batch to many.
> Will propose a fix.

Here's a standalone reproducer with the right parameters to reach this
error, and a simple fix. (Definitely room for more improvements in
this area of code... but that'll have to be a project for later.)

===8<===
shared_buffers=2GB
fsync=off
max_wal_size=10GB
min_dynamic_shared_memory=2GB
===8<===
create table bigger_than_it_looks as
select generate_series(1, 256000000) as id;
alter table bigger_than_it_looks set (autovacuum_enabled = 'false');
alter table bigger_than_it_looks set (parallel_workers = 1);
analyze bigger_than_it_looks;
update pg_class set reltuples = 5000000 where relname = 'bigger_than_it_looks';
===8<===
postgres=# set work_mem = '4.5GB';
SET
postgres=# explain analyze select count(*) from bigger_than_it_looks
t1 join bigger_than_it_looks t2 using (id);
ERROR: invalid DSA memory alloc request size 1073741824
CONTEXT: parallel worker
===8<===