From: | Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> |
---|---|
To: | andres(at)anarazel(dot)de |
Cc: | ah(at)cybertec(dot)at, magnus(at)hagander(dot)net, robertmhaas(at)gmail(dot)com, tgl(at)sss(dot)pgh(dot)pa(dot)us, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: shared-memory based stats collector |
Date: | 2018-09-27 13:00:49 |
Message-ID: | 20180927.220049.168546206.horiguchi.kyotaro@lab.ntt.co.jp |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Hello. This is a super-PoC of no-UDP stats collector.
At Wed, 26 Sep 2018 09:55:09 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote in <20180926(dot)095509(dot)182252925(dot)horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
> > I don't think either of these is right. I think it's crucial to get rid
> > of the UDP socket, but I think using a shmem queue is the wrong
> > approach. Not just because postgres' shm_mq is single-reader/writer, but
> > also because it's plainly unnecessary. Backends should attempt to
> > update the shared hashtable, but acquire the necessary lock
> > conditionally, and leave the pending updates of the shared hashtable to
> > a later time if they cannot acquire the lock.
>
> Ok, I just intended to avoid reading many bytes from a file and
> thought that writer-side can be resolved later.
>
> Currently locks on the shared stats table is acquired by dshash
> mechanism in a partition-wise manner. The number of the
> partitions is currently fixed to 2^7 = 128, but writes for the
> same table confilicts each other regardless of the number of
> partitions. As the first step, I'm going to add
> conditional-locking capability to dsahsh_find_or_insert and each
> backend holds a queue of its pending updates.
I don't have more time 'til next monday so this is just a PoC
(sorry..).
- 0001 to 0006 is rebased version of v4.
- 0007 adds conditional locking to dshash
- 0008 is the no-UDP stats collector.
If required lock is not acquired for some stats items, report
funcions immediately return after storing the values locally. The
stored values are merged with later calls. Explicitly calling
pgstat_cleanup_pending_stat() at a convenient timing tries to
apply the pending values, but the function is not called anywhere
for now.
stats collector process is used only to save and load saved stats
files and create shared memory for stats. I'm going to remove
stats collector.
I'll continue working this way.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachment | Content-Type | Size |
---|---|---|
v5-0008-Ultra-PoC-of-full-shared-memory-stats-collector.patch | text/x-patch | 92.5 KB |
v5-0007-Add-conditional-lock-feature-to-dshash.patch | text/x-patch | 4.1 KB |
v5-0006-Remove-pg_stat_tmp-exclusion-from-pg_rewind.patch | text/x-patch | 1.1 KB |
v5-0005-Let-pg_stat_statements-not-to-use-PG_STAT_TMP_DIR.patch | text/x-patch | 1.9 KB |
v5-0004-Documentation-update.patch | text/x-patch | 6.1 KB |
v5-0003-dshash-based-stats-collector.patch | text/x-patch | 93.7 KB |
v5-0002-Change-stats-collector-to-an-axiliary-process.patch | text/x-patch | 13.3 KB |
v5-0001-sequential-scan-for-dshash.patch | text/x-patch | 6.8 KB |
From | Date | Subject | |
---|---|---|---|
Next Message | Dean Rasheed | 2018-09-27 13:22:39 | Re: BUG #15307: Low numerical precision of (Co-) Variance |
Previous Message | Michael Paquier | 2018-09-27 12:51:55 | Re: Problem while setting the fpw with SIGHUP |