Re: strange nbtree corruption report

On Mon, Nov 21, 2011 at 08:00:21PM -0300, Alvaro Herrera wrote:
> We got a very strange nbtree corruption report some time ago. This was
> a btree index on a vey high churn table -- entries are updated and
> deleted very quickly, so the index grows very large and also shrinks
> quickly (AFAICT this is a work queue of sorts).
>
> The most strange thing of all is that there was this error:
>
> ERROR: left link changed unexpectedly in block 3378 of index "index_name"
> CONTEXT: automatic vacuum of table "table_name"

Just a suspicion ... when looking at the B-tree page reclamation algorithm, I
had a thought that the logic in _bt_page_recyclable() was obsolete as of the
introduction (in 8.3) of xid-free read-only transactions. A transaction
without a persistent xid does not hold back RecentXmin, so how could waiting
for a RecentXmin window to pass prove that no scan still holds a link to the
page? Similarly, running VACUUMs do not hold back RecentXmin. I had made a
note to try to reproduce a concrete problem, but I haven't tried yet.

> This was reported not once, but several dozens of times, by each new
> autovacuum worker that tried to vacuum the table.
>
> As far as I can see, there is just no way for this to happen ... much
> less happen repeatedly. I thought it might be related to concurrent
> insertions somehow managing to split the page under deletion very
> quickly (given the load these systems are under, this is plausible).
> But I can't find how.

Yes, nothing comes to mind explaining that duration of persistence.

nm