Re: (resolution?) Re: [HACKERS] memory problem again

Tom,

I think the #2 is more likely, because:

the kernel is compiled with large enough data size:

# support for larger processes and number of childs
options "DFLDSIZ=\(128*1024*1024\)"
options "MAXDSIZ=\(256*1024*1024\)"
options "CHILD_MAX=256"
options "OPEN_MAX=256"
options "KMAPENTRIES=4000" # Prevents kmem malloc errors !
options "KMEMSIZE=\(32*1024*1024\)"

the default postgres acocunt limits are:

coredumpsize unlimited
cputime unlimited
datasize 131072 kbytes
filesize unlimited
maxproc 256
memorylocked 85380 kbytes
memoryuse 256136 kbytes
openfiles 128
stacksize 2048 kbytes

I run the postmaster after unlimit, which sets limits thus:

coredumpsize unlimited
cputime unlimited
datasize 262144 kbytes
filesize unlimited
maxproc 4116
memorylocked 256140 kbytes
memoryuse 256136 kbytes
openfiles 13196
stacksize 262144 kbytes

I will do some experimentation with the -S flag to see how it works.

BTW, this postgres is compiled with default of 64 backends - I saw recently
note here that this may interfere with the -S option somehow....

(another not related bug, but still on memory allocation)
Still - this does not explain why postgres cannot allocated more than 76 MB
(according to top) on BSD/OS (never did, actually - any previous version too),
while a simple malloc(1 MB) loop allocates up to the process limit.

Maybe at some time postrges tries to allocate 'larger' chunk, which the BSD/OS
malloc does not like?

Daniel

>>>Tom Lane said:
> Daniel Kalchev <daniel(at)digsys(dot)bg> writes:
> > I found out how to resolve this problem, yet it does not explain why it
> > happens anyway!
> > I had postmaster started with this script:
> > postmaster -D/usr/local/pgsql/data -B 256 -i -o "-e -S 8192" >>
> > /usr/local/pgsql/errlog 2>&1 &
> > Removing all the parameters to postmaster
> > postmaster -D/usr/local/pgsql/data -i -o "-e" >> /usr/local/pgsql/errlog 2
>&1 &
> > made it work....
> > Perhaps some memory management problem? I guess the -S option is the culpr
it
> > here, but this machine has 256 MB RAM and actually never swaps (yet).
>
> 8192 * 1K = 8 meg workspace per sort sure doesn't sound unreasonable.
> There is a sort going on under-the-hood in your SELECT DISTINCT (it's
> implemented in the same fashion as "sort | uniq"), but under ordinary
> circumstances that doesn't cause any problem. I can see a couple of
> possibilities:
> 1. You have a very small kernel limit on per-process data space,
> probably 8M or at most 16M.
> 2. Something is broken in the sort code that makes it fail to
> obey the -S limit.
> I favor #1, since if #2 were true we'd probably have noticed it before.
>
> You might try experimenting with a couple of different -S values (-B
> shouldn't make any difference here, it just affects the size of the
> shared-memory-block request), and watching the size of the backend
> process with top(1) or something like it.
>
> In the meantime, find out where kernel parameters are set on your
> system, and look at what MAXDSIZ is set to...
>
> regards, tom lane