Re: shared-memory based stats collector

Hello.

At Tue, 02 Oct 2018 16:06:51 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote in <20181002(dot)160651(dot)117284090(dot)horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
> It doesn't work nor even compile since I failed to include some
> changes. The atached v6-0008 at least compiles and words.
>
> 0001-0007 are not attached since they are still aplicable on
> master head with offsets.

In this patchset 0001-0007 are still the same with the previous
version. I'll reorganize the whole patchset in the next version.

This is more saner version of previous v5-0008, which didn't pass
regression test. v6-0008 to v6-0010 are attached and they are
applied on top of v5-0001-0007.

- stats collector has been removed.

- modified dshash further so that deletion is allowed during
sequential scan.

- I'm not sure about the following existing comment at the
beginning of pgstat.c

* - Add a pgstat config column to pg_database, so this
* entire thing can be enabled/disabled on a per db basis.

Some points known to need some considerations are:

1. Concurrency is controlled by per-database entry in db_stats
dshash. It has 127 lock partitions but all backends on the
same database share just one lock and only one backend takes
the right to update stats. (Every backend doesn't update stats
with the interval not shorter than 500ms, like the current
stats collector.) Table-stats can be removed by DROP DB
simultaeously with stats updates so need to block it using
per-databsae lock. Any locking means other than dshash might
be needed.

2. Since dshash cannot allow multiple locks because of resize,
pgstat_update_stat is forced to be a bit ineffecient.
It loops onver stats list twice, for shared tables and regular
tables since we can acquire lock on one database at once.

Maybe providing individual TabStatusArray for the two will fix
it, will do in the next version.

3. This adds a new timeout IDLE_STATS_UPDATE_TIMEOUT. This works
similarly to IDLE_IN_TRANSACTIION_SESSION_TIMEOUT. It fires in
at most PGSTAT_STAT_MIN_INTERVAL(500)ms to clean up pending
statistics update.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center