Re: BUG #16980: invalid memory alloc request size

Michael Paquier <michael(at)paquier(dot)xyz> writes:
> On Fri, Apr 23, 2021 at 10:10:39AM +0000, PG Bug reporting form wrote:
>> I have a table attachment_contents in my PostgreSQL database with bytea type
>> column. One cell of this bytea column is too big (appr. 900Mb).
>> I receive error when try to select this cell:
>> taxmon=> select content from attachment_contents where attachment_id = 3293;
>> ERROR: invalid memory alloc request size 1850079891

>> How can i avoid this problem?

> No idea without more context. Some of your data may be corrupted,
> leading to this error. It may also be possible that you have
> discovered a new bug, but it is going to be hard for someone to figure
> out what's happening without a self-contained test case.

I doubt there's any corruption or unknown bug. If the bytea itself
is 900MB, its hex-string representation will be twice that, exceeding
the 1GB palloc chunk-size limit. So we can't represent the value as
text, and even if we could, it wouldn't fit into a DataRow message.

You can imagine various hacks that would move the threshold for
trouble, but the long and the short of it is that field values
exceeding maybe 100MB are likely to cause problems somewhere.
If you have to store such things, you'd be best off to keep them
in files outside the database.

regards, tom lane