Re: PostgreSQL 8.0.6 crash

On Thu, Feb 09, 2006 at 02:03:41PM -0500, Mark Woodward wrote:
> > "Mark Woodward" <pgsql(at)mohawksoft(dot)com> writes:
> >> Again, regardless of OS used, hashagg will exceed "working memory" as
> >> defined in postgresql.conf.
> >
> > So? If you've got OOM kill enabled, it can zap a process whether it's
> > strictly adhered to work_mem or not. The OOM killer is entirely capable
> > of choosing a victim process whose memory footprint hasn't changed
> > materially since it started (eg, the postmaster).
>
> Sorry, I must strongly disagree here. The postgresql.conf "working mem" is
> a VERY IMPORTANT setting, it is intended to limit the consumption of
> memory by the postgresql process. Often times PostgreSQL will work along

Actually, no, it's not designed for that at all.

> side other application servers on the same system, infact, may be a
> sub-part of application servers on the same system. (This is, in fact, how
> it is used on one of my site servers.)
>
> Clearly, if the server will use 1000 times this number (Set for 1024K, but
> exceeds 1G) this is broken, and it may cause other systems to fail or
> perform very poorly.
>
> If it is not something that can be fixed, it should be clearly documented.

work_mem (integer)

Specifies the amount of memory to be used by internal sort
operations and hash tables before switching to temporary disk files.
The value is specified in kilobytes, and defaults to 1024 kilobytes
(1 MB). Note that for a complex query, several sort or hash
operations might be running in parallel; each one will be allowed to
use as much memory as this value specifies before it starts to put
data into temporary files. Also, several running sessions could be
doing such operations concurrently. So the total memory used could
be many times the value of work_mem; it is necessary to keep this
fact in mind when choosing the value. Sort operations are used for
ORDER BY, DISTINCT, and merge joins. Hash tables are used in hash
joins, hash-based aggregation, and hash-based processing of IN
subqueries.

So it says right there that it's very easy to exceed work_mem by a very
large amount. Granted, this is a very painful problem to deal with and
will hopefully be changed at some point, but it's pretty clear as to how
this works.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461