Re: postgresql +AMD64 +big address spaces - does it work?

On Thu, 2004-07-01 at 06:41, Andy B wrote:
> If I bought one of these boxes/OS combos as a postgresql database server,
> would postgresql be able to make the best use of it with a huge (e.g. 40GB)
> database?
>
> Box: HP ProLiant DL585, with 4 AMD64 CPUs and 64GB of RAM. (other
> vendor options also exist)
> OS: SUSE enterprise 8 linux for AMD
>
> (links to product info at bottom)
>
> e.g. Could postgresql make use of most of this address space to map a huge
> database (say 40GB) all paged into RAM at once?
>
> ----
> According to Suse, a process running on such a setup can individually
> address 0.5 terabytes of address space, and at any one point, with the
> hardware above, up to 64GB (minus a bit!) of this address space could be
> paged into physical memory at any one time.
> ----
>
>
> According to the docs I could find, I just need to do the following:
>
> 1. set postgresql.conf->shared_buffers to a value like (40GB/8KB),
>
> 2. Check cat /proc/sys/kernel/shmmax is big enough, or successfully increase
> it to > 40GB
>
> >From experience that's a long way from knowing it will work. Does anyone
> have any experience at all with such big memory configurations?

Sort of. PostgreSQL could use however much of it you decided to give it
as a buffer for individual queries, and the newer caching algo should
handle large amounts of data well.

However, PostgreSQL isn't currently in the business, so to speak, of
holding all the data you've accessed in the last X minutes in a great
big buffer. The reason for this is that PostgreSQL uses shared memory,
and on many architectures, shared memory is good for letting individual
backends communicate quickly, it's not as fast at managing really huge
chunks of data.

So, PostgreSQL is designed to let your kernel do all this. The buffers
in PostgreSQL are designed to hold the current working set, no more.
The buffers in the kernel cache are designed to handle gigs of data, and
to handle it quickly. So, if it's in Kernel cache, then PostgreSQL IS
capable of utilizing the extra memory, in the sense that the kernel is
using it for cache.

All modern flavors of unix have pretty good caching built in, with
little tuning needed.