Re: BUG #16925: ERROR: invalid DSA memory alloc request size 1073741824 CONTEXT: parallel worker

On Sat, Mar 13, 2021 at 11:21 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
> On Sat, Mar 13, 2021 at 8:12 AM PG Bug reporting form
> <noreply(at)postgresql(dot)org> wrote:
> > Using TPC-H benchmark scale-factor 1000. To recreate dataset, see
> > https://github.com/swarm64/s64da-benchmark-toolkit. Use psql_native
> > schema.
>
> I'd like to reproduce this but it may take me some time. Can you
> please show the query plan?
>
> > 2021-03-12 19:45:37.352 CET [316243] ERROR: XX000: invalid DSA memory alloc
> > request size 1073741824
>
> So, this means we have a call to dsa_allocate_extended() without the
> DSA_ALLOC_HUGE flag failing the sanity check that surely no one wants
> a GB of memory at once. In hash joins, we deliberately avoid making
> our hash table bucket array long enough to hit that, since commit
> 86a2218e, and all other data is allocated in small chunks. So the
> only way to hit this would be with an individual tuple that takes 1GB
> to store. Other paths that use DSA include bitmap heap scans, but
> they use the DSA_ALLOC_HUGE flag at least in one place.
>
> > max_parallel_workers | 1000
> > max_parallel_workers_per_gather | 52

Another way to make a very large single allocation with a parallel
hash join with a large number of partitions and participants. Are you
in a position to debug and test patched versions? It'd be interesting
to know if ExecParallelHashJoinSetUpBatches() is the location, where
it does:

pstate->batches =
dsa_allocate0(hashtable->area,
EstimateParallelHashJoinBatch(hashtable) * nbatch);

If so, questions of whether it's really sane to run with so many
batches aside, the solution could be dsa_allocate_extended(...,
DSA_ALLOC_HUGE | DSA_ALLOC_ZERO).