Re: select on 22 GB table causes "An I/O error occured while sending to the backend." exception

On Thu, 28 Aug 2008, david(at)lang(dot)hm wrote:

> On Thu, 28 Aug 2008, Matthew Wakeling wrote:
>
>> On Wed, 27 Aug 2008, david(at)lang(dot)hm wrote:
>>> if memory overcommit is disabled, the kernel checks to see if you have an
>>> extra 1G of ram available, if you do it allows the process to continue, if
>>> you don't it tries to free memory (by throwing away cache, swapping to
>>> disk, etc), and if it can't free the memory will return a memroy
>>> allocation error (which I believe will cause firefox to exit).
>>
>> Remember that the memory overcommit check is checking against the amount of
>> RAM + swap you have - not just the amount of RAM. When a fork occurs,
>> hardly any extra actual RAM is used (due to copy on write), but the
>> potential is there for the process to use it. If overcommit is switched
>> off, then you just need to make sure there is *plenty* of swap to convince
>> the kernel that it can actually fulfil all of the memory requests if all
>> the processes behave badly and all shared pages become unshared. Then the
>> consequences of processes actually using that memory are that the machine
>> will swap, rather than the OOM killer having to act.
>
> if you are correct that it just checks against memory+swap then it's not a
> big deal, but I don't think it does that. I think it actually allocates the
> memory, and if it does that it will push things out of ram to do the
> allocation, I don't believe that it will allocate swap space directly.

I just asked on the kernel mailing list and Alan Cox responded.

he is saying that you are correct, it only allocates against the total
available, it doesn't actually allocate ram.

so with sufficiant swap overcommit off should be fine.

but you do need to allocate more swap as the total memory 'used' can be
significantly higher that with overcommit on.

David Lang

> David Lang
>
>> Of course, it's generally bad to run a machine with more going on than will
>> fit in RAM.
>>
>> Neither swapping nor OOM killing are particularly good - it's just a
>> consequence of the amount of memory needed being unpredictable.
>>
>> Probably the best solution is to just tell the kernel somehow to never kill
>> the postmaster.
>>
>> Matthew
>>
>>
>