Re: Large PostgreSQL servers

On Thu, Mar 22, 2012 at 10:57 AM, Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com> wrote:
> On Thu, Mar 22, 2012 at 9:29 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
>> On Thu, Mar 22, 2012 at 10:02 AM, Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com> wrote:
>>> There's other issues you run into with large shared_buffers as well.
>>> If you've got a large shared_buffers setting, but only regularly hit a
>>> small subset of your db (say 32GB shared_buffers but only hit 4G or so
>>> regularly in your app) then it's quite possible that older
>>> shared_buffer segments will get swapped out because they're not being
>>> used.  Then, when the db goes to hit a page in shared_buffers, the OS
>>> will have to swap it back in.  What was supposed to make your db much
>>> faster has now made it much slower.
>>>
>>> With Linux, the OS tends to swap out unused memory to make room for
>>> file buffers.  While you can change the swappiness settings to 0 to
>>> slow it down, the OS will eventually swap out the least used segments
>>> anyway.  The only solution on large memory servers is often to just
>>> turn off swap.
>>
>> Right -- but my take on that is that hacking the o/s to disable swap
>> is dealing with symptoms of problem related to server
>> misconfiguration.
>
> You can configure a big memory linux server anyway you want.  After a
> while, they seem to go crazy anyway and start swapping even when
> you've told them not to.
>
>> In particular it probably means shared_buffers is set too high...the
>> o/s thinks it needs that memory more than you do and it may very well
>> be right.
>
> I've had machines with 128GB RAM and a 4G shared_buffers start
> swapping for no apparent reason and just fall over.  There's no memory
> pressure etc, just kswapd decides to go nuts and start swapping.
>
> This was on Ubuntu 10.04 and 12.04 and RHEL 5.2 through 5.latest with
> all updates.  These machines typically had ~90GB+ of kernel cache and
> zero memory pressure.

hm, that's interesting -- noted. I'll keep an eye out for that.

merlin