Re: BUG #18775: PQgetCopyData always has an out-of-memory error if the table field stores bytea ~700 MB

From: Ilya Knyazev <knuazev(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: pgsql-bugs(at)lists(dot)postgresql(dot)org
Subject: Re: BUG #18775: PQgetCopyData always has an out-of-memory error if the table field stores bytea ~700 MB
Date: 2025-01-16 17:02:03
Message-ID: CAFqQ4ZWyUwwwgHA+BBGiLKbmm+SaRPJZQSgOHajDa+AB+igfRA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

Thanks for the answer. I agree that if you want to work with large data,
then make sure you have enough memory. If you call the PQexec function with
a regular select and then PQgetvalue to get the value. And you get an error
- then it's your own fault.
But I know that there may not be enough memory, so I use the "copy" keyword
in the query and the PQgetCopyData function. I thought that this function
was designed for portioned work. By analogy with the PQputCopyData
function, which works fine.

I found a way out. I use "SELECT substring(%s from %d for %d) as chunk from
%s.%s WHERE %s = %s" with PQunescapeBytea in a loop. But it looks strange
and takes longer.

p .s
I specified that my file is 300+ MB, not 700+.

чт, 16 янв. 2025 г. в 19:03, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:

> PG Bug reporting form <noreply(at)postgresql(dot)org> writes:
> > Hello, I need your help. The essence of the problem is that I am trying
> to
> > download a bytea field from a table row. The size of the data in the
> field
> > is about 700 MB. In response, I receive an out-of-memory error.
>
> I don't see this as a Postgres bug. If you want to work with values
> that large, you'd better have plenty of memory available.
>
> regards, tom lane
>

--
С уважением,
Князев И. Н.

In response to

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message PG Bug reporting form 2025-01-16 18:35:54 BUG #18777: Error running unnest function in a two phase commit transaction
Previous Message Tom Lane 2025-01-16 16:02:59 Re: BUG #18775: PQgetCopyData always has an out-of-memory error if the table field stores bytea ~700 MB