Re: LOG: could not fork new process for connection: Cannot allocate memory

Hi John,

Thanks for replying. Below are all the details

I am using psql (PostgreSQL) 9.5.2

**** Error we got in postgresql log*

2016-08-25 11:15:55 PDT [60739]: [10282-1] LOG: could not fork new process
for connection: Cannot allocate memory
2016-08-25 11:15:55 PDT [60739]: [10283-1] LOG: could not fork new process
for connection: Cannot allocate memory
2016-08-25 11:15:55 PDT [60739]: [10284-1] LOG: could not fork new process
for connection: Cannot allocate memory
2016-08-25 11:15:55 PDT [60739]: [10285-1] LOG: could not fork new process
for connection: Cannot allocate memory
2016-08-25 11:15:55 PDT [60739]: [10286-1] LOG: could not fork new process
for connection: Cannot allocate memory
2016-08-25 11:15:55 PDT [60739]: [10287-1] LOG: could not fork new process
for connection: Cannot allocate memory

**** OS*

Red Hat Enterprise Linux Server release 6.3 (Santiago)
I dont see any errors in /var/log/messages

**** OS Configuration*

-bash-4.1$ cat /etc/sysctl.conf
#ipv4 definitions

net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

#begin=

vm.swappiness=10
vm.overcommit_memory=2
vm.overcommit_ratio=85
vm.dirty_background_ratio=1
vm.dirty_ratio=20
vm.dirty_expire_centisecs=500
vm.dirty_writeback_centisecs=100
vm.zone_reclaim_mode=0
vm.nr_hugepages=25600
vm.hugetlb_shm_group=26
vm.nr_overcommit_hugepages=512
vm.dirty_background_ratio=1

kernel.shmmax=214748364800
kernel.shmall=52428800
kernel.sem = 4010 1002500 4010 350

fs.file-max=1000000
net.ipv4.conf.all.log_martians = 0

*** Memory output

-bash-4.1$ cat /proc/meminfo
MemTotal: 264493868 kB
MemFree: 6158268 kB
Buffers: 64584 kB
Cached: 170097488 kB
SwapCached: 928152 kB
Active: 164001880 kB
Inactive: 57038336 kB
Active(anon): 80960992 kB
Inactive(anon): 3445472 kB
Active(file): 83040888 kB
Inactive(file): 53592864 kB
Unevictable: 4996 kB
Mlocked: 4996 kB
SwapTotal: 8388592 kB
SwapFree: 5251268 kB
Dirty: 253228 kB
Writeback: 5116 kB
AnonPages: 49991856 kB
Mapped: 32847112 kB
Shmem: 33523772 kB
Slab: 1987048 kB
SReclaimable: 1664692 kB
SUnreclaim: 322356 kB
KernelStack: 23912 kB
PageTables: 18320680 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 221273452 kB
Committed_AS: 95458972 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 688676 kB
VmallocChunk: 34224374380 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 6856
HugePages_Free: 6836
HugePages_Rsvd: 59
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7852 kB
DirectMap2M: 3102720 kB
DirectMap1G: 265289728 kB

-bash-4.1$ free -m -g
total used free shared buffers cached
Mem: 252 248 3 0 0 164
-/+ buffers/cache: 84 167
Swap: 7 2 5

*** Kernel settings

# sudo -iu postgres ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2066203
max locked memory (kbytes, -l) 209715200
max memory size (kbytes, -m) unlimited
open files (-n) 1000000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 5500
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

cat /etc/security/limits.conf

postgres soft nproc 5500
postgres hard nproc 5500
postgres soft nofile 1000000
postgres hard nofile 1000000
postgres soft memlock 209715200
postgres hard memlock 209715200

$ grep Commit /proc/meminfo
CommitLimit: 133625220 kB
Committed_AS: 82635628 kB

$ cat /proc/sys/vm/overcommit_memory
2

*** DB Parameter settings
spool of database parameter file is
https://paste.fedoraproject.org/414084/56014147/

Regards
Ali

On Thu, Aug 25, 2016 at 12:57 PM, John R Pierce <pierce(at)hogranch(dot)com> wrote:

> On 8/25/2016 11:49 AM, Ahsan Ali wrote:
>
>>
>> I having my production server service intruption because of this "Cannot
>> allocate memory" error in the db log.
>>
>
> could you paste the whole error message?
>
> what version of postgres is this?
>
> what OS (if linux, distribution) version is this?
>
>
> older versions of postgres require kernel.shmmax and some other settings
> to be increased if you request larger shared_memory settings. this is
> likely what you're running into.
>
>
> --
> john r pierce, recycling bits in santa cruz
>
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>