From: | Michael Paquier <michael(at)paquier(dot)xyz> |
---|---|
To: | Postgres hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
Subject: | WAL-logging facility for pgstats kinds |
Date: | 2024-12-27 03:32:25 |
Message-ID: | Z24fyb3ipXKR38oS@paquier.xyz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Hi all,
While brainstorming on the contents of the thread I have posted a
couple of days ago, I have been looking at what could be done so as
pgstats and WAL-logging could work together. This was point 2) from
this message:
https://www.postgresql.org/message-id/Z2tblEmfuOfZy4zx@paquier.xyz
While considering the point that not all stats data is worth
replicating, I have fallen down to the following properties that are
nice to have across the board:
- A pgstats kind should be able to WAL-log some data that is should be
able to decide. Including versioning of the data.
- The data kind should be able to decide how this data is handled at
recovery (different persistency depending on standby vs crash
recovery, for one).
- Having to add one RMGR for each stats kind is not going to scale,
especially as we can control the redo path using a callback part of
PgStat_KindInfo. For custom kinds, this enforces some validation
based on if the stats library has been really loaded at startup with
shared_preload_library.
- It is nice for each out-of-core stats kind to not have to load and
define a custom RMGR, duplicating what this central facility is doing.
- The persistency of the stats data is better across crashes: this
approach makes the cleanup of orphaned pgstats entries easier as this
can be enforced at replay by each stats kind, if necessary, and it
is also possible to force the creation of stats data.
- For variable-numbered stats, the WAL-logging can be timed with the
timing where any pending stats are flushed, if there is some.
Table stats to prevent autovacuum from committing Seppuku for tables
without stats after a crash is the first use case I can think about,
where we'd want to split and expose some DML stats. Scans stats
should remain untouched on standbys, for example. There may be other
stats data that is worth replicating, all these could use this new
facility.
Attached is the result of this investigation, where I have finished by
implementing a new backend facility where pgstats kinds can WAL-log
data at will using a new RMGR called "PgStat". The most interesting
part is pgstat_xlog_data() in a new file called pgstat_xlog.c, that
stats kinds can use as an entry point to WAL-log data structures with
a XLogRegisterData() managed by a new record structure called
xl_pgstat_data.
For clarity, the patch set has been split into several pieces, I hope
rather edible:
- 0001, a fix I've posted on a different thread [1], used in patch
0004 to test this new facility.
- 0002, a refactoring piece to be able to expose stats kind data into
the frontend (for pg_waldump).
- 0003 is the core backend piece, introducing the WAL-logging routine
and the new RMGR.
- 0004, that provides tests for the new backend facility via a custom
stats kind. Requires 0001.
I am going to attach it to the next CF.
Comments or opinions are welcome.
[1]: https://www.postgresql.org/message-id/Z23zcE4w1djukkva@paquier.xyz
--
Michael
Attachment | Content-Type | Size |
---|---|---|
v1-0001-injection_points-Tweak-variable-numbered-stats-to.patch | text/x-diff | 2.5 KB |
v1-0002-Move-information-about-pgstats-kinds-into-its-own.patch | text/x-diff | 5.7 KB |
v1-0003-Add-RMGR-and-WAL-logging-API-for-pgstats.patch | text/x-diff | 11.2 KB |
v1-0004-injection_points-Add-option-and-tests-for-WAL-log.patch | text/x-diff | 7.4 KB |
From | Date | Subject | |
---|---|---|---|
Next Message | vignesh C | 2024-12-27 03:52:18 | Re: Introduce XID age and inactive timeout based replication slot invalidation |
Previous Message | wenhui qiu | 2024-12-27 02:04:48 | Re: PoC: history of recent vacuum/checkpoint runs (using new hooks) |