Re: select on 22 GB table causes "An I/O error occured while sending to the backend." exception

On Thu, 28 Aug 2008, Matthew Wakeling wrote:

> On Wed, 27 Aug 2008, david(at)lang(dot)hm wrote:
>> if memory overcommit is disabled, the kernel checks to see if you have an
>> extra 1G of ram available, if you do it allows the process to continue, if
>> you don't it tries to free memory (by throwing away cache, swapping to
>> disk, etc), and if it can't free the memory will return a memroy allocation
>> error (which I believe will cause firefox to exit).
>
> Remember that the memory overcommit check is checking against the amount of
> RAM + swap you have - not just the amount of RAM. When a fork occurs, hardly
> any extra actual RAM is used (due to copy on write), but the potential is
> there for the process to use it. If overcommit is switched off, then you just
> need to make sure there is *plenty* of swap to convince the kernel that it
> can actually fulfil all of the memory requests if all the processes behave
> badly and all shared pages become unshared. Then the consequences of
> processes actually using that memory are that the machine will swap, rather
> than the OOM killer having to act.

if you are correct that it just checks against memory+swap then it's not a
big deal, but I don't think it does that. I think it actually allocates
the memory, and if it does that it will push things out of ram to do the
allocation, I don't believe that it will allocate swap space directly.

David Lang

> Of course, it's generally bad to run a machine with more going on than will
> fit in RAM.
>
> Neither swapping nor OOM killing are particularly good - it's just a
> consequence of the amount of memory needed being unpredictable.
>
> Probably the best solution is to just tell the kernel somehow to never kill
> the postmaster.
>
> Matthew
>
>