Re: pg_dump out of memory for large table with LOB

From: Adrien Nayrat <adrien(dot)nayrat(at)anayrat(dot)info>
To: <pgsql-general(at)lists(dot)postgresql(dot)org>
Subject: Re: pg_dump out of memory for large table with LOB
Date: 2018-11-11 10:14:44
Message-ID: 73d2f4fb-3da4-1c09-8bff-88a35f16bcd2@anayrat.info
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hello,

On 11/10/18 12:49 AM, Jean-Marc Lessard wrote:
> The dumped table is 0.5TB, 17 million rows and LOB uses about 99% of the space.
>

If I understand, you have 17 million Large Object?

I do not recall exactly and maybe I am wrong. But it seems pg_dump has to
allocate memory for each object to dump :
addBoundaryDependencies:

for (i = 0; i < numObjs; i++)
[...]

case DO_BLOB_DATA:
/* Data objects: must come between the boundaries */
addObjectDependency(dobj, preDataBound->dumpId);
addObjectDependency(postDataBound, dobj->dumpId);
break;

addObjectDependency:

[...]
pg_malloc(dobj->allocDeps * sizeof(DumpId));

With 17 million LO, it could eat lot of memory ;)

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Rob Nikander 2018-11-11 16:20:23 query patterns for multipass aggregating
Previous Message Adrian Klaver 2018-11-10 23:31:31 Re: pg_dump out of memory for large table with LOB