Re: BUG #8034: pg_buffercache gets invalid memory alloc request size with very large shared memory buffers

On 03/04/13 08:44, dbenhur(at)whitepages(dot)com wrote:
> The following bug has been logged on the website:
>
> Bug reference: 8034
> Logged by: Devin Ben-Hur
> Email address: dbenhur(at)whitepages(dot)com
> PostgreSQL version: 9.2.3
> Operating system: Ubuntu Precise
> Description:
>
> When a very large shared buffer pool (~480GB) is used with postgresql,
> pg_buffercache contrib module gets an allocation error trying to Allocate
> NBuffers worth of BufferCachePagesRec records:
>
> https://github.com/postgres/postgres/blob/REL9_2_3/contrib/pg_buffercache/pg_buffercache_pages.c#L101-L102
>
> The requested allocation exceeds the 1GB limitation imposed by
> AllocSizeIsValid macro:
> https://github.com/postgres/postgres/blob/REL9_2_3/src/include/utils/memutils.h#L40-L43
>
> Reproduce:
> 1) acquire server with half terabyte of memory
> 2) tweak OS settings to allow large shared memory
> 3) set postgresql.conf: shared_buffers = 400GB
> 4) CREATE EXTENSION pg_buffercache;
> 5) SELECT * FROM pg_buffercache LIMIT 1;
>
>
>

Yes indeed - however I'm not sure this is likely to be encountered in
any serious configuration. The general rule for sizing shared buffers is:

shared_buffers = min(0.25 * RAM, 8G)

Now there has been some discussion about how settings bigger than 8G
make sense in some cases...but I'm not aware of any suggestions that
sizes in the hundreds of G make sense.

However it would be nice if pg_buffercache *could* work with bigger
sizes if they make sense at any time. Someone who understands the
memory allocation system better than I do will need to comment about how
that might work :-)

Cheers

Mark