Re: pg_restore out of memory

Gregory Stark writes:

> You're right that your limit is above 128M but the error just means it tried
> to allocated 128M and failed, it may already have allocated 400M and been
> trying to go over the 524M mark.

My limit should be 1.6GB as per /boot/loader.conf

I ran a ps every 20 seconds to try and catch the program as it was going
over the size.

The ps right before the crash is:
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
pgsql 9364 6.2 22.8 904472 803436 ?? Rs
8:34PM 139:37.46 postgres: pgsql backaway_copy [local] COPY (postgres)

fran 9363 8.2 8.6 338928 305332 p3 S+
8:34PM 244:49.41 pg_restore -U pgsql -v -d backaway_copy backaway-fc-2007-06-03.pg_d

pgsql 15373 2.3 10.6 613892 374000 ?? Rs 3:06AM
11:45.28 postgres: pgsql full_table_restores [local] VACUUM (postgres)

904472 = 883MB
So if it was trying to allocate 128MB more it crashed trying to allocate
somewhere around 1011 MB. Although since I do it every 20 seconds.. could
have gone beyond that.

The question is, what type of memory is that trying to allocate?
Shared memory?
Shared memory from is kern.ipc.shmmax=534773760, well below the 1011MB or
above that I am estimating it crashed from.

The FreeBSD default is to cap programs at 512MB so thet fact the program is
crashing at 1011MB means that the /boot/loader.conf setting of 1.6GB memory
cap is active.

If it is of any help.. information about the data.
It is hex64 encoded data. The original data getting encoded can be up to
100MB. Not sure how much bigger the hex encoding could be making it.