Re: ERROR: out of memory | with 23GB cached 7GB reserved on 30GB machine

I didn't realize that about overcommit_ratio. It was at 50, I've changed it
to 95. I'll see if that clears up the problem moving forward.

# cat /proc/meminfo
MemTotal: 30827220 kB
MemFree: 153524 kB
MemAvailable: 17941864 kB
Buffers: 6188 kB
Cached: 24560208 kB
SwapCached: 0 kB
Active: 20971256 kB
Inactive: 8538660 kB
Active(anon): 12460680 kB
Inactive(anon): 36612 kB
Active(file): 8510576 kB
Inactive(file): 8502048 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 50088 kB
Writeback: 160 kB
AnonPages: 4943740 kB
Mapped: 7571496 kB
Shmem: 7553176 kB
Slab: 886428 kB
SReclaimable: 858936 kB
SUnreclaim: 27492 kB
KernelStack: 4208 kB
PageTables: 188352 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 15413608 kB
Committed_AS: 14690544 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 59012 kB
VmallocChunk: 34359642367 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 31465472 kB
DirectMap2M: 0 kB

# sysctl -a:

vm.admin_reserve_kbytes = 8192

vm.block_dump = 0

vm.dirty_background_bytes = 0

vm.dirty_background_ratio = 10

vm.dirty_bytes = 0

vm.dirty_expire_centisecs = 3000

vm.dirty_ratio = 20

vm.dirty_writeback_centisecs = 500

vm.drop_caches = 0

vm.extfrag_threshold = 500

vm.hugepages_treat_as_movable = 0

vm.hugetlb_shm_group = 0

vm.laptop_mode = 0

vm.legacy_va_layout = 0

vm.lowmem_reserve_ratio = 256 256 32

vm.max_map_count = 65530

vm.min_free_kbytes = 22207

vm.min_slab_ratio = 5

vm.min_unmapped_ratio = 1

vm.mmap_min_addr = 4096

vm.nr_hugepages = 0

vm.nr_hugepages_mempolicy = 0

vm.nr_overcommit_hugepages = 0

vm.nr_pdflush_threads = 0

vm.numa_zonelist_order = default

vm.oom_dump_tasks = 1

vm.oom_kill_allocating_task = 0

vm.overcommit_kbytes = 0

vm.overcommit_memory = 2

vm.overcommit_ratio = 50

vm.page-cluster = 3

vm.panic_on_oom = 0

vm.percpu_pagelist_fraction = 0

vm.scan_unevictable_pages = 0

vm.stat_interval = 1

vm.swappiness = 0

vm.user_reserve_kbytes = 131072

vm.vfs_cache_pressure = 100

vm.zone_reclaim_mode = 0

On Tue, Oct 21, 2014 at 3:46 PM, Tomas Vondra <tv(at)fuzzy(dot)cz> wrote:
>
> Dne 22 Říjen 2014, 0:25, Montana Low napsal(a):
> > I'm running postgres-9.3 on a 30GB ec2 xen instance w/ linux kernel
> > 3.16.3.
> > I receive numerous Error: out of memory messages in the log, which are
> > aborting client requests, even though there appears to be 23GB available
> > in
> > the OS cache.
> >
> > There is no swap on the box. Postgres is behind pgbouncer to protect
from
> > the 200 real clients, which limits connections to 32, although there are
> > rarely more than 20 active connections, even though postgres
> > max_connections is set very high for historic reasons. There is also a
4GB
> > java process running on the box.
> >
> >
> >
> >
> > relevant postgresql.conf:
> >
> > max_connections = 1000 # (change requires restart)
> > shared_buffers = 7GB # min 128kB
> > work_mem = 40MB # min 64kB
> > maintenance_work_mem = 1GB # min 1MB
> > effective_cache_size = 20GB
> >
> >
> >
> > sysctl.conf:
> >
> > vm.swappiness = 0
> > vm.overcommit_memory = 2
>
> This means you have 'no overcommit', so the amount of memory is limited by
> overcommit_ratio + swap. The default value for overcommit_ratio is 50%
> RAM, and as you have no swap that effectively means only 50% of the RAM is
> available to the system.
>
> If you want to verify this, check /proc/meminfo - see the lines
> CommitLimit (the current limit) and Commited_AS (committed address space).
> Once the committed_as reaches the limit, it's game over.
>
> There are different ways to fix this, or at least improve that:
>
> (1) increasing the overcommit_ratio (clearly, 50% is way too low -
> something 90% might be more appropriate on 30GB RAM without swap)
>
> (2) adding swap (say a small ephemeral drive, with swappiness=10 or
> something like that)
>
> Tomas
>