Re: relfrozenxid may disagree with row XIDs after 1ccc1e05ae

Hi,

On 2024-05-16 12:39:53 -0700, Andres Freund wrote:
> On 2024-05-16 12:49:00 -0400, Peter Geoghegan wrote:
> > It looked very much like index vacuuming was involved in some way when
> > I actually had the opportunity to use gdb against an affected
> > production instance that ran into the problem.
>
> Hm, were the cases you observed that way using parallel vacuuming? And what
> index types were involved?
>
> Melanies reproducer works because there are catalog accesses that can trigger
> a recomputation of fuzzy horizon. For testing the "easy" window for that is
> the vac_open_indexes() < 16, because it happens after determining the horizon,
> but before actually vacuuming.
>
> Now I wonder if there is some codepath triggering catalog lookups during bulk
> delete.

There is in gin:

2024-05-16 13:02:56.114 PDT [1632692][client backend][2/4:0][psql] CONTEXT: while scanning block 33 of relation "public.foo"
2024-05-16 13:02:56.114 PDT [1632692][client backend][2/4:0][psql] STATEMENT: VACUUM (VERBOSE, PARALLEL 0) foo;
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
1632692: AIM
2024-05-16 13:02:56.116 PDT [1632261][postmaster][:0][] DEBUG: server process (PID 1632484) exited with exit code 0
2024-05-16 13:02:56.140 PDT [1632692][client backend][2/4:0][psql] DEBUG: scanned index "foo_gin" to remove 7458 row versions

#0 AcceptInvalidationMessages () at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/inval.c:808
#1 0x0000000000ce1342 in LockRelationOid (relid=1255, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:136
#2 0x000000000081b718 in relation_open (relationId=1255, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/common/relation.c:55
#3 0x00000000008d8116 in table_open (relationId=1255, lockmode=1) at ../../../../../home/andres/src/postgresql/src/backend/access/table/table.c:44
#4 0x0000000000ec8b04 in SearchCatCacheMiss (cache=0x289f180, nkeys=1, hashValue=3157698538, hashIndex=106, v1=16641, v2=0, v3=0, v4=0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1468
#5 0x0000000000ec8a80 in SearchCatCacheInternal (cache=0x289f180, nkeys=1, v1=16641, v2=0, v3=0, v4=0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1413
#6 0x0000000000ec8773 in SearchCatCache1 (cache=0x289f180, v1=16641) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1281
#7 0x0000000000ee59cf in SearchSysCache1 (cacheId=45, key1=16641) at ../../../../../home/andres/src/postgresql/src/backend/utils/cache/syscache.c:225
#8 0x0000000000ef6496 in fmgr_info_cxt_security (functionId=16641, finfo=0x28e7dd8, mcxt=0x28fa5b0, ignore_security=false)
at ../../../../../home/andres/src/postgresql/src/backend/utils/fmgr/fmgr.c:181
#9 0x0000000000ef63cb in fmgr_info_cxt (functionId=16641, finfo=0x28e7dd8, mcxt=0x28fa5b0)
at ../../../../../home/andres/src/postgresql/src/backend/utils/fmgr/fmgr.c:139
#10 0x0000000000894ee7 in index_getprocinfo (irel=0x7f53fae86e60, attnum=1, procnum=2)
at ../../../../../home/andres/src/postgresql/src/backend/access/index/indexam.c:902
#11 0x000000000083d962 in initGinState (state=0x7ffc7c077960, index=0x7f53fae86e60)
at ../../../../../home/andres/src/postgresql/src/backend/access/gin/ginutil.c:155
#12 0x000000000083ff1c in ginbulkdelete (info=0x7ffc7c07a010, stats=0x0, callback=0xa559ca <vac_tid_reaped>, callback_state=0x29036d8)
at ../../../../../home/andres/src/postgresql/src/backend/access/gin/ginvacuum.c:582
#13 0x00000000008949bd in index_bulk_delete (info=0x7ffc7c07a010, istat=0x0, callback=0xa559ca <vac_tid_reaped>, callback_state=0x29036d8)
at ../../../../../home/andres/src/postgresql/src/backend/access/index/indexam.c:758
#14 0x0000000000a55872 in vac_bulkdel_one_index (ivinfo=0x7ffc7c07a010, istat=0x0, dead_items=0x29036d8, dead_items_info=0x29036b8)
at ../../../../../home/andres/src/postgresql/src/backend/commands/vacuum.c:2498
#15 0x000000000088e94c in lazy_vacuum_one_index (indrel=0x7f53fae86e60, istat=0x0, reltuples=40000, vacrel=0x2902770)
at ../../../../../home/andres/src/postgresql/src/backend/access/heap/vacuumlazy.c:2451
#16 0x000000000088dc94 in lazy_vacuum_all_indexes (vacrel=0x2902770) at ../../../../../home/andres/src/postgresql/src/backend/access/heap/vacuumlazy.c:2032
#17 0x000000000088d9e4 in lazy_vacuum (vacrel=0x2902770) at ../../../../../home/andres/src/postgresql/src/backend/access/heap/vacuumlazy.c:1946
#18 0x000000000088c2e9 in lazy_scan_heap (vacrel=0x2902770) at ../../../../../home/andres/src/postgresql/src/backend/access/heap/vacuumlazy.c:897

Greetings,

Andres Freund