From: | Justin Lu <justin(dot)lu5432(at)gmail(dot)com> |
---|---|
To: | Andres Freund <andres(at)anarazel(dot)de> |
Cc: | pgsql-admin(at)lists(dot)postgresql(dot)org, pgsql-admin(at)postgresql(dot)org |
Subject: | Re: Heavy LWLockTranche buffer_mapping in Postgres 9.6.10 server |
Date: | 2020-02-03 19:30:55 |
Message-ID: | CACnXL0DroJ7VmTPsmRaSmJ8Ra7XWGR3ne=RZdmx3D8sO72PTbA@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-admin |
Thanks for those pitched in. I finally resolved the issue. It turns out
that the table bloat was the culprit. Those bloat couldn't be dealt with by
manual or auto vacuum. It somehow reached a point that many queries had
trouble finish on time even though they used good query plans. The final
solution was just do a full vacuum on some of the key tables, and that
immediately quiet down the database.
thanks,
Justin
On Sun, Feb 2, 2020, 8:15 AM Andres Freund <andres(at)anarazel(dot)de> wrote:
> Hi,
>
> On 2020-02-01 16:17:13 -0700, Justin Lu wrote:
> > We are seeing very heavy LWLockTranche buffer_mapping in db recently.
> >
> > There server had 24 core, 128GB of RAM, SSD data file system, on Unbuntu
> > 16.04.6.
> > The shared_buffers was at 32GB. 1/4 of over RAM size. No issue on
> > checkpoints (avg time 29 min apart).
> >
> > After seeing the heavy wait, we added 64GB more RAM and increased
> > shared_buffers to 48GB, effective_cache_size to 90GB. But it seems there
> is
> > no impact on the buffer mapping waits at all.
>
> I suggest doing a perf profile with --call-graph dwarf, to see where
> this is mostly coming from.
>
> One thing I've seen causing symptoms like this before, is if there's
> suddenly a larger amount of table truncations, dropping, etc - dropping
> / truncating a table / index needs to scan all of shared buffers...
>
> Greetings,
>
> Andres Freund
>
From | Date | Subject | |
---|---|---|---|
Next Message | Sterpu Victor | 2020-02-06 09:03:28 | Error "is not a table or materialized view" when creating a unique index on a materialized view on PostgreSQL 9.5.10 |
Previous Message | Andres Freund | 2020-02-02 15:15:03 | Re: Heavy LWLockTranche buffer_mapping in Postgres 9.6.10 server |