Re: postgres vacuum memory limits

On Sun, 1 Aug 2021 at 20:04, Vijaykumar Jain <
vijaykumarjain(dot)github(at)gmail(dot)com> wrote:

>
> On Sun, 1 Aug 2021 at 10:27, Ayub M <hiayub(at)gmail(dot)com> wrote:
> >
> > Hello, when maintenance_work_mem and autovacuum_work_mem are set, my
> understanding is that the vacuum and autovacuum sessions should be limited
> to use the memory limits set by these parameters. But I am seeing more
> memory being used than these limits by autovacuum sessions, any reason why
> this would happen?
> >
> > Please see below examples, where maintenance_work_mem is set to 20mb and
> shared_buffers is 128mb. When I see the memory for this session in top, it
> shows 162mb. But when default_statistics_target is increased to 3000, the
> session usage is 463mb, which is way more than 20mb maintenance_work_mem
> and 128mb shared_buffer. Shouldn't the process memory be capped to 20+128mb?
> >
> >
> > postgres=# show maintenance_work_mem ;
> > maintenance_work_mem
> > ----------------------
> > 20MB
> > (1 row)
> > postgres=# vacuum analyze mdm_context;
> > VACUUM
> > postgres=# show shared_buffers;
> > shared_buffers
> > ----------------
> > 128MB
> > (1 row)
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
>
> > 62246 postgres 20 0 422892 165996 139068 R 57.1 15.7 25:06.34
> postgres: postgres postgres [local] VACUUM
> >
>
> your assumption may be right, but i am not sure of the interpretation from
> top.
> I have to admit I am not great at understanding top output (RES vs VIRT)
> in general when it comes to limits.
>
>
i just tried a couple of more things, maybe it helps.

every connection ( like psql in interactive mode ) has an overhead of
around 10MB.

postgres=# set maintenance_work_mem TO 1024;
SET
postgres=# -- 11284 this was showing in my RES mem on a fresh connection
postgres=# do $$
begin
for i in 1..20 loop
update t set col1 = col1 || i::text;
commit;
end loop;
end; $$;
DO
postgres=# -- 394924 this was showing in my RES mem on a connection that
did large updates, adding to connection cache ?
postgres=# vacuum t;
VACUUM
postgres=# -- 395852 this was showing in my RES mem on a connection that
did vacuum, although the value is around 395M,
-- but vacuum only took around ~ 1M when
maintenance_work_mem was set to 1024 (1MB)

PostgreSQL connections are process based, and a lot goes into what is held
into the memories right since init, i did a pmap and lsof on the process id,
it touches a lot of files in datadir/base and datadir/global, basically the
pages touched during the session activities.

also there is a huge chunk allocated to
00007f233b839000 2164816K rw-s- /dev/zero (deleted)
which I think is mmap to /dev/zero that contents have been deleted, but the
connection has to be closed to reclaim that space. idk