Re: BUG #8034: pg_buffercache gets invalid memory alloc request size with very large shared memory buffers

Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> writes:
>> When a very large shared buffer pool (~480GB) is used with postgresql,
>> pg_buffercache contrib module gets an allocation error trying to Allocate
>> NBuffers worth of BufferCachePagesRec records:

> Yes indeed - however I'm not sure this is likely to be encountered in
> any serious configuration.

I too am a bit skeptical of trying to make this actually work. For one
thing, pg_buffercache would be locking down the entire buffer arena for
a rather significant amount of time while it transfers gigabytes of data
into the local array. What would likely make more sense, if we ever get
to the point where this is a practical size of configuration, is to
provide a mechanism to read out data for just a portion of the arena
at a time.

> However it would be nice if pg_buffercache *could* work with bigger
> sizes if they make sense at any time. Someone who understands the
> memory allocation system better than I do will need to comment about how
> that might work :-)

There has been some discussion of inventing a "big_palloc"
infrastructure to allow allocation of arrays larger than 1GB, for use
in places like the sort code. If we ever get around to doing that,
it'd be straightforward enough to make pg_buffercache use the facility
... but I really doubt pg_buffercache itself is a sufficient reason
to do it.

regards, tom lane