Re: Memory reporting on CentOS Linux

If I have 10GB of ram, and I see a process using 5Gb of RES size. Then TOP should at least report 5GB physical memory used (AKA: Not available in CACHED, or FREE). If I run a 'free -m', I should only see 5GB of ram available. I can understand with virtual memory that some of it may be on disk, therefore I may not see this memory being taken away from the physical memory available.

I am thoroughly confused that TOP is reporting that I have 99% of my physical RAM free, while the process list suggests that some are taking ~8Gb of Resident (Physical) Memory. Any explanation as to why TOP is reporting this? I have a PostgreSQL 8.3 server with 48Gb of RAM on a Dell R610 server that is reporting that 46.5GB of RAM is free. This confuses me to no end. Why is it not reporting much more physical memory used?

[root(at)pg6 jcarroll]# free -m
total used free shared buffers cached
Mem: 48275 48136 138 0 141 46159
-/+ buffers/cache: 1835 46439
Swap: 2047 12 2035
Thanks!

top - 09:24:38 up 17:05, 1 user, load average: 1.09, 1.08, 1.18
Tasks: 239 total, 2 running, 237 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.2%us, 0.1%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49433916k total, 49295460k used, 138456k free, 145308k buffers
Swap: 2097144k total, 12840k used, 2084304k free, 47267056k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13200 postgres 15 0 16.1g 15g 15g R 2.3 33.6 2:36.78 postgres: writer process
29029 postgres 25 0 16.1g 13g 13g R 99.9 29.4 36:35.13 postgres: dbuser database 192.168.200.8(36979) UPDATE
13198 postgres 15 0 16.1g 317m 316m S 0.0 0.7 0:00.57 /opt/PostgreSQL/8.3/bin/postgres -D /opt/PostgreSQL/8.3/data
13201 postgres 15 0 16.1g 2300 1824 S 0.0 0.0 0:00.39 postgres: wal writer process
13202 postgres 15 0 98.7m 1580 672 S 0.0 0.0 0:15.12 postgres: stats collector process

-----Original Message-----
From: Scott Carey [mailto:scott(at)richrelevance(dot)com]
Sent: Friday, August 14, 2009 3:38 PM
To: Jeremy Carroll; pgsql-performance(at)postgresql(dot)org
Subject: Re: [PERFORM] Memory reporting on CentOS Linux

On 8/14/09 11:00 AM, "Jeremy Carroll" <jeremy(dot)carroll(at)networkedinsights(dot)com>
wrote:

> I am confused about what the OS is reporting for memory usage on CentOS 5.3
> Linux. Looking at the resident memory size of the processes. Looking at the
> resident size of all postgres processes, the system should be using around
> 30Gb of physical ram. I know that it states that it is using a lot of shared
> memory. My question is how to I determine how much physical RAM postgres is
> using at any point in time?

Resident includes Shared. Shared is shared. So you have to subtract it
from all the processes to see what they use on their own. What you really
want is RES-SHR, or some of the other columns available in top. Hit 'h' in
top to get some help on the other columns available, and 'f' and 'o'
manipulate them. In particular, you might find the "DATA" column useful.
It is approximately RES-SHR-CODE

>
> This server has 24Gb of ram, and is reporting that 23GB is free for use. See
> calculation below
>
> (Memory Total Used) + (Buffers + Cached) = Free Memory
> (24675740 24105052) + (140312 + 22825616) = 23,536,616 or ~23 Gigabytes
>
>
> So if my server has 23Gb of ram that is free for use, why is postgres
> reporting resident sizes of 30GB? Shared memory is reporting the same values,
> so how is the OS reporting that only 1Gb of RAM is being used?
>
> Help?
>
> top - 12:43:41 up 2 days, 19:04, 2 users, load average: 4.99, 4.81, 4.33
> Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> Cpu(s): 26.0%us, 0.0%sy, 0.0%ni, 73.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 24675740k total, 24105052k used, 570688k free, 140312k buffers
> Swap: 2097144k total, 272k used, 2096872k free, 22825616k cached
> ---------------------
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 19469 postgres 15 0 8324m 7.9g 7.9g S 0.0 33.7 0:54.30 postgres: writer
> process
> 29763 postgres 25 0 8329m 4.5g 4.5g R 99.8 19.0 24:53.02 postgres: dbuser
> database x.x.x.49(51136) UPDATE

Lets just take the two above and pretend that they are the only postgres
processes.
The RAM used by each exclusively is RES-SHR. Or, close to nothing for these
two, aside from the rounding error.

The memory used by postgres for shared memory is the largest of all SHR
columns for postgres columns. Or, about 7.9GB. So, postgres is using
about 7.9GB for shared memory, and very little for anything else.

In formula form, its close to
SUM(RES) - SUM(SHR) + MAX(SHR).
That doesn't cover everything, but is very close. See the other columns
available in top.