From: | Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com> |
---|---|
To: | Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> |
Cc: | Bernd Helmle <bernd(at)oopsware(dot)de>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: 9.3.9 and pg_multixact corruption |
Date: | 2015-09-10 23:51:08 |
Message-ID: | CAEepm=2xW7L-etbyEVEOspDSwDFmSvwR_=EO_duE2USNFvO5xA@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Fri, Sep 11, 2015 at 10:45 AM, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>
wrote:
> Bernd Helmle wrote:
> > A customer had a severe issue with a PostgreSQL 9.3.9/sparc64/Solaris 11
> > instance.
> >
> > The database crashed with the following log messages:
> >
> > 2015-09-08 00:49:16 CEST [2912] PANIC: could not access status of
> > transaction 1068235595
> > 2015-09-08 00:49:16 CEST [2912] DETAIL: Could not open file
> > "pg_multixact/members/FFFF5FC4": No such file or directory.
> > 2015-09-08 00:49:16 CEST [2912] STATEMENT: delete from StockTransfer
> > where oid = $1 and tanum = $2
>
> I wonder if these bogus page and offset numbers are just
> SlruReportIOError being confused because pg_multixact/members is so
> weird (I don't think it should be the case, since this stuff is using
> page numbers only, not anything related to how each page is layed out).
>
But SlruReportIOError uses the same macro to build the filename as
SlruReadPhysicalPage and other functions, namely SlruFileName which uses
sprintf with %04X (unsigned integer uppercase hex) and gives it segno
(which is always an int), so I don't think the problem is in error
reporting only.
Assuming default block size, to get FFFF5FC4 from SlruFileName you need
segno == -41020.
We have int segno = pageno / 32 (that's SLRU_PAGES_PER_SEGMENT), so to get
segno == -41020 you need pageno between -1312640 and -1312609 (whose bit
patterns reinterpreted as unsigned are 4293654656 and 4293654687).
In various places we have int pageno = offset / (uint32) 1636, expanded
from this macro (which calls the offset an xid):
#define MXOffsetToMemberPage(xid) ((xid) / (TransactionId)
MULTIXACT_MEMBERS_PER_PAGE)
I don't really see how any uint32 value could produce such a pageno via
that macro. Even if called in an environment where (xid) is accidentally
an int, the int / unsigned expression would convert it to unsigned first
(unless (xid) is a bigger type like int64_t: by the rules of int promotion
you'd get signed division in that case, hmm...). But it's always called
with a MultiXactOffset AKA uint32 variable.
So via that route, there is no MultiXactOffset value that can't be mapped
to a segment in the range "0000", "14078". Famously, it wraps after that.
Maybe the negative pageno came from somewhere else. Where? Inside SLRU
code we can see pageno = shared->page_number[slotno]... maybe the SLRU
slots got corrupted somehow?
--
Thomas Munro
http://www.enterprisedb.com
From | Date | Subject | |
---|---|---|---|
Next Message | Thomas Munro | 2015-09-10 23:58:48 | Re: 9.3.9 and pg_multixact corruption |
Previous Message | Костя Кузнецов | 2015-09-10 22:52:30 | New gist vacuum. |