reduce overhead in shared memory TID store

Going from one to two vacuum workers doesn't scale as well as we'd
like, and this thread is about ways of mitigating that.

The easiest thing that shows some benefit is separating the two paths
by preventing inlining of one or the other:

#define RT_PREFIX shared_ts
#define RT_SHMEM
-#define RT_SCOPE static
+#define RT_SCOPE static pg_noinline

This small change is enough to show a difference in an unrealistic
test designed to reduce I/O and overemphasize TID storage and
retrieval:

0001: a patch from Masahiko to time the vacuum stages separately
0002: force non-parallel vacuum to use shared memory for easier testing
0003: the above 1-line change

drop table if exists test;
create unlogged table test (a int) with (AUTOVACUUM_ENABLED=false);
insert into test select * from generate_series(1,20_000_000,1);
create index on test (a);

select pg_prewarm('test');
select pg_prewarm('test_a_idx');

delete from test where a < 20_000_000;
vacuum verbose test;

Two runs each:

0002:

local:
INFO: heap_scan 1731 index_vac 586 heap_vac 119
INFO: heap_scan 1772 index_vac 592 heap_vac 119
shared:
INFO: heap_scan 2059 index_vac 894 heap_vac 121
INFO: heap_scan 2070 index_vac 893 heap_vac 120

0003:

local:
INFO: heap_scan 1669 index_vac 543 heap_vac 119
INFO: heap_scan 1670 index_vac 543 heap_vac 119
shared:
INFO: heap_scan 1978 index_vac 839 heap_vac 120
INFO: heap_scan 1989 index_vac 838 heap_vac 119

This makes both paths faster, and is pretty good for a 1-line change,
so I intend to commit 0003. In the real world, with disk and WAL, this
would probably only be noticeable on tables with many indexes, but
I've encountered plenty of those.

Note that phase III is unaffected, and I think it's because the
iteration state relies on saved local pointers. We could expand that
concept, but it'd be invasive and unreliable.

There are other things we can try, and I'll update the thread as I find them.

--
John Naylor
Amazon Web Services