Re: Invalid memory alloc request size for repeat()

On Thu, 26 May 2022 at 09:03, Japin Li <japinli(at)hotmail(dot)com> wrote:
> On Wed, 25 May 2022 at 22:50, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Japin Li <japinli(at)hotmail(dot)com> writes:
>>> Today, I try to use repeat() to generate 1GB text, and it occurs invalid memory
>>> alloc request size [1]. It is a limit from palloc(), then I try to reduce it,
>>> it still complains out of memory which comes from enlargeStringInfo() [2]. The
>>> documentation about repect() [3] doesn't mentaion the limitation.
>>
>> It would probably make sense for repeat() to check this explicitly:
>>
>> if (unlikely(pg_mul_s32_overflow(count, slen, &tlen)) ||
>> - unlikely(pg_add_s32_overflow(tlen, VARHDRSZ, &tlen)))
>> + unlikely(pg_add_s32_overflow(tlen, VARHDRSZ, &tlen)) ||
>> + unlikely(!AllocSizeIsValid(tlen)))
>> ereport(ERROR,
>> (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
>> errmsg("requested length too large")));
>>
>
> LGTM. Thanks for your patch!

After some analysis, I found it might not easy to solve this.

For example,

```
postgres=# CREATE TABLE myrepeat AS SELECT repeat('a', 1024 * 1024 * 1024 - 5);
ERROR: invalid memory alloc request size 1073741871
```

Here is the backtrace:

#0 palloc0 (size=1073741871) at /mnt/workspace/postgresql/build/../src/backend/utils/mmgr/mcxt.c:1103
#1 0x0000561925199faf in heap_form_tuple (tupleDescriptor=0x561927cb4310, values=0x561927cb4470, isnull=0x561927cb4478)
at /mnt/workspace/postgresql/build/../src/backend/access/common/heaptuple.c:1069
#2 0x00005619254879aa in tts_virtual_copy_heap_tuple (slot=0x561927cb4428) at /mnt/workspace/postgresql/build/../src/backend/executor/execTuples.c:280
#3 0x000056192548a40c in ExecFetchSlotHeapTuple (slot=0x561927cb4428, materialize=true, shouldFree=0x7ffc9cc5197f)
at /mnt/workspace/postgresql/build/../src/backend/executor/execTuples.c:1660
#4 0x000056192520e0c9 in heapam_tuple_insert (relation=0x7fdacaa9d3e0, slot=0x561927cb4428, cid=5, options=2, bistate=0x561927cb83f0)
at /mnt/workspace/postgresql/build/../src/backend/access/heap/heapam_handler.c:245
#5 0x00005619253b2edf in table_tuple_insert (rel=0x7fdacaa9d3e0, slot=0x561927cb4428, cid=5, options=2, bistate=0x561927cb83f0)
at /mnt/workspace/postgresql/build/../src/include/access/tableam.h:1376
#6 0x00005619253b3d73 in intorel_receive (slot=0x561927cb4428, self=0x561927c82430) at /mnt/workspace/postgresql/build/../src/backend/commands/createas.c:586
#7 0x0000561925478d86 in ExecutePlan (estate=0x561927cb3ed0, planstate=0x561927cb4108, use_parallel_mode=false, operation=CMD_SELECT, sendTuples=true, numberTuples=0,
direction=ForwardScanDirection, dest=0x561927c82430, execute_once=true) at /mnt/workspace/postgresql/build/../src/backend/executor/execMain.c:1667
#8 0x0000561925476735 in standard_ExecutorRun (queryDesc=0x561927cab990, direction=ForwardScanDirection, count=0, execute_once=true)
at /mnt/workspace/postgresql/build/../src/backend/executor/execMain.c:363
#9 0x000056192547654b in ExecutorRun (queryDesc=0x561927cab990, direction=ForwardScanDirection, count=0, execute_once=true)
at /mnt/workspace/postgresql/build/../src/backend/executor/execMain.c:307
#10 0x00005619253b3711 in ExecCreateTableAs (pstate=0x561927c35b00, stmt=0x561927b5aa70, params=0x0, queryEnv=0x0, qc=0x7ffc9cc52370)

The heap_form_tupe() function need extra 48 bytes for HeapTupleHeaderData and HEAPTUPLESIZE.

If we use the following, everything is ok.

postgres=# CREATE TABLE myrepeat AS SELECT repeat('a', 1024 * 1024 * 1024 - 5 - 48);
SELECT 1

If we want to send the result to the client. Here is an error.

postgres=# SELECT repeat('a', 1024 * 1024 * 1024 - 5);
ERROR: out of memory
DETAIL: Cannot enlarge string buffer containing 6 bytes by 1073741819 more bytes.

This is because the printtup() needs send to the number of attributes (2-byte)
and the length of data (4-byte) and then the data (1073741819-byte).

Do those mean we cannot store 1GB to a field [1] and send 1GB of data to the client?

[1] https://www.postgresql.org/docs/current/limits.html

--
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.