Unexplained Access Exclusive Locks on Follower from PID7

From: Andy Cooper <andrew(dot)cooper(at)salesforce(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: Unexplained Access Exclusive Locks on Follower from PID7
Date: 2020-05-15 21:16:38
Message-ID: CACom7q8dT3GvsDHuyEKONL1Xc0wimf-Mh4HbqyTnXrW-Ut2baA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

We recently saw a follower database taken down as the result of maxed out
connection slots.

The logs showed that the lock was held by PID 7 and was blocking an
AccessShareLock

[9-1] sql_error_code = 00000 LOG: process 5148 still waiting for
AccessShareLock on relation 2840 of database 16402 after 1000.103 ms
9-2] sql_error_code = 00000 DETAIL: Process holding the lock: 7. Wait
queue: ...

Given that I did not catch the locking in progress I am having to rely on
these logs to determine what occurred. The waiting for an AccessShareLock
leads me to believe that the actual lock must have been AccessExclusive.

Relation 2840 is a TOAST table whose primary table is `pg_statistic`.

One other piece of potential evidence was that there were auto vacuums
occurring on the primary around this time.

Outside of the lock logs there is no logging around what PID7 was doing.
Additionally the other followers in the formation did not suffer from the
same locking.

Any ideas on what this might be or how we could further troubleshoot this
issue?

--
Andy Cooper

Browse pgsql-general by date

  From Date Subject
Next Message Christophe Pettus 2020-05-15 22:07:42 Re: Inherited an 18TB DB & need to backup
Previous Message Adrian Klaver 2020-05-15 19:57:00 Re: Bug on version 12 ?