Re: Some other CLOBBER_CACHE_ALWAYS culprits

Hi,

On 2021-05-11 19:02:00 -0700, Andres Freund wrote:
> I don't think we'd loose a lot of practical coverage if we avoided
> rebuilding non-accessed relcache entries eagerly during cache
> lookups. What coverage do we e.g. gain by having a single
> SearchCatCacheMiss() triggering rebuilding the relcache of a user
> defined table several times?
>
> The InvalidateSystemCaches() marks all catcache entries as invalid. The
> next catcache lookup will thus trigger a cache miss. That cache miss
> will typically at least open the previously not locked relation + index
> the cache is over. Each of those relation opens will fire off another
> InvalidateSystemCaches(). Which will rebuild all the surviving relcache
> entries at least twice - despite never being accessed in that path.

This is actually worse than I described here, and I think it may point
towards a relatively minimal change that'd improve performance of
debug_invalidate_system_caches_always=1 substantially.

Putting in some instrumentation I noticed that with
debug_invalidate_system_caches_always=1 a single "top level"
SearchCatCacheMiss() triggers up to a 100 RelationCacheInvalidate(). There's
two levels to it:

The table_open/index_open done as part of a SearchCatCacheMiss() will each
trigger a invalidation of their own. But what then drives that up much further
is that the RelationCacheInvalidate() will destroy the relcache entries for
nearly all indexes and for pg_amop etc and *not* rebuild them as part of
RelationCacheInvalidate() - there are no references.

Which means that the index_open() on whatever index the syscache uses builds a
new relache entry. Which then needs to do a RelationInitIndexAccessInfo() on
that index. Which triggers a lot of syscache lookups. Which in turn need to
build pg_omop etc. Which trigggers RelationCacheInvalidate() over and over.

In essence, debug_invalidate_system_caches_always=1 in some important aspects
behaves like debug_invalidate_system_caches_always=3, due to the syscache
involvement.

I think it's worth testing that we actually deal with everything possible
being invalidated as part of a syscache lookup, but I don't think we learn a
ton doing that for the whole build. Particularly when it prevents us from
actually testing more interesting invalidation scenarios?

What about having a mode where each "nesting" level of SearchCatCacheMiss
allows only one interior InvalidateSystemCaches()?

Here's an example stacktrace showing three nested syscache lookups:

#0 SearchCatCacheMiss (cache=0x55aed2c34e00, nkeys=2, hashValue=3953514454, hashIndex=86, v1=2656, v2=2, v3=0, v4=0)
at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1329
#1 0x000055aed0edf194 in SearchCatCacheInternal (cache=0x55aed2c34e00, nkeys=2, v1=2656, v2=2, v3=0, v4=0)
at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1301
#2 0x000055aed0edeeb8 in SearchCatCache2 (cache=0x55aed2c34e00, v1=2656, v2=2) at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1177
#3 0x000055aed0efc9b6 in SearchSysCache2 (cacheId=7, key1=2656, key2=2) at /home/andres/src/postgresql/src/backend/utils/cache/syscache.c:1145
#4 0x000055aed0ee46aa in get_attoptions (relid=2656, attnum=2) at /home/andres/src/postgresql/src/backend/utils/cache/lsyscache.c:1002
#5 0x000055aed0ef7ebd in RelationGetIndexAttOptions (relation=0x7f873ad21700, copy=false)
at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:5734
#6 0x000055aed0eefc95 in RelationInitIndexAccessInfo (relation=0x7f873ad21700) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1522
#7 0x000055aed0eee927 in RelationBuildDesc (targetRelId=2656, insertIt=true) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1194
#8 0x000055aed0ef09ea in RelationIdGetRelation (relationId=2656) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:2064
#9 0x000055aed083a95d in relation_open (relationId=2656, lockmode=1) at /home/andres/src/postgresql/src/backend/access/common/relation.c:59
#10 0x000055aed08c77af in index_open (relationId=2656, lockmode=1) at /home/andres/src/postgresql/src/backend/access/index/indexam.c:136
#11 0x000055aed08c6be4 in systable_beginscan (heapRelation=0x7f873ad1ec60, indexId=2656, indexOK=true, snapshot=0x0, nkeys=1, key=0x7ffdff557420)
at /home/andres/src/postgresql/src/backend/access/index/genam.c:395
#12 0x000055aed0ef436b in AttrDefaultFetch (relation=0x7f873ad1e830, ndef=1) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:4422
#13 0x000055aed0eed6dc in RelationBuildTupleDesc (relation=0x7f873ad1e830) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:689
#14 0x000055aed0eee737 in RelationBuildDesc (targetRelId=16385, insertIt=false) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1147
#15 0x000055aed0ef16e4 in RelationClearRelation (relation=0x7f873ad1c728, rebuild=true)
at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:2592
#16 0x000055aed0ef2391 in RelationCacheInvalidate () at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:3047
#17 0x000055aed0ee2218 in InvalidateSystemCaches () at /home/andres/src/postgresql/src/backend/utils/cache/inval.c:657
#18 0x000055aed0ee230b in AcceptInvalidationMessages () at /home/andres/src/postgresql/src/backend/utils/cache/inval.c:725
#19 0x000055aed0d35204 in LockRelationOid (relid=2610, lockmode=1) at /home/andres/src/postgresql/src/backend/storage/lmgr/lmgr.c:137
#20 0x000055aed083a953 in relation_open (relationId=2610, lockmode=1) at /home/andres/src/postgresql/src/backend/access/common/relation.c:56
#21 0x000055aed0913b73 in table_open (relationId=2610, lockmode=1) at /home/andres/src/postgresql/src/backend/access/table/table.c:43
#22 0x000055aed0edf2be in SearchCatCacheMiss (cache=0x55aed2c3dc80, nkeys=1, hashValue=1574576467, hashIndex=19, v1=2696, v2=0, v3=0, v4=0)
at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1365
#23 0x000055aed0edf194 in SearchCatCacheInternal (cache=0x55aed2c3dc80, nkeys=1, v1=2696, v2=0, v3=0, v4=0)
at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1301
#24 0x000055aed0edee7d in SearchCatCache1 (cache=0x55aed2c3dc80, v1=2696) at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1169
#25 0x000055aed0efc8dd in SearchSysCache1 (cacheId=32, key1=2696) at /home/andres/src/postgresql/src/backend/utils/cache/syscache.c:1134
#26 0x000055aed0eeef67 in RelationInitIndexAccessInfo (relation=0x7f873ad1e160) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1401
#27 0x000055aed0eee927 in RelationBuildDesc (targetRelId=2696, insertIt=true) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:1194
#28 0x000055aed0ef09ea in RelationIdGetRelation (relationId=2696) at /home/andres/src/postgresql/src/backend/utils/cache/relcache.c:2064
#29 0x000055aed083a95d in relation_open (relationId=2696, lockmode=1) at /home/andres/src/postgresql/src/backend/access/common/relation.c:59
#30 0x000055aed08c77af in index_open (relationId=2696, lockmode=1) at /home/andres/src/postgresql/src/backend/access/index/indexam.c:136
#31 0x000055aed08c6be4 in systable_beginscan (heapRelation=0x7f873ad21040, indexId=2696, indexOK=true, snapshot=0x0, nkeys=3, key=0x7ffdff557f60)
at /home/andres/src/postgresql/src/backend/access/index/genam.c:395
#32 0x000055aed0edf30f in SearchCatCacheMiss (cache=0x55aed2c49380, nkeys=3, hashValue=1153660433, hashIndex=17, v1=16385, v2=1, v3=0, v4=0)
at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1367
#33 0x000055aed0edf194 in SearchCatCacheInternal (cache=0x55aed2c49380, nkeys=3, v1=16385, v2=1, v3=0, v4=0)
at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1301
#34 0x000055aed0edeef8 in SearchCatCache3 (cache=0x55aed2c49380, v1=16385, v2=1, v3=0)
at /home/andres/src/postgresql/src/backend/utils/cache/catcache.c:1185
#35 0x000055aed0efca94 in SearchSysCache3 (cacheId=59, key1=16385, key2=1, key3=0) at /home/andres/src/postgresql/src/backend/utils/cache/syscache.c:1156
#36 0x000055aed0ee75eb in get_attavgwidth (relid=16385, attnum=1) at /home/andres/src/postgresql/src/backend/utils/cache/lsyscache.c:3116

Greetings,

Andres Freund