Re: Pluggable cumulative statistics

On Thu, Jun 20, 2024 at 01:05:42PM +0000, Bertrand Drouvot wrote:
> On Thu, Jun 13, 2024 at 04:59:50PM +0900, Michael Paquier wrote:
>> * Making custom stats data persistent is an interesting problem, and
>> there are a couple of approaches I've considered:
>> ** Allow custom kinds to define callbacks to read and write data from
>> a source they'd want, like their own file through a fd. This has the
>> disadvantage to remove the benefit of c) above.
>> ** Store everything in the existing stats file, adding one type of
>> entry like 'S' and 'N' with a "custom" type, where the *name* of the
>> custom stats kind is stored instead of its ID computed from shared
>> memory.
>
> What about having 2 files?
>
> - One is the existing stats file
> - One "predefined" for all the custom stats (so what you've done minus the
> in-core stats). This one would not be configurable and the extensions will
> not need to know about it.

Another thing that can be done here is to add a few callbacks to
control how an entry should be written out when the dshash is scanned
or read when the dshash is populated depending on the KindInfo.
That's not really complicated to do as the populate part could have a
cleanup phase if an error is found. I just did not do it yet because
this patch set is already covering a lot, just to get the basics in.

> Would that remove the benefit from c) that you mentioned up-thread?

Yes, that can be slightly annoying. Splitting the stats across
multiple files would mean that each stats file would have to store the
redo LSN. That's not really complicated to implement, but really easy
to miss. Perhaps folks implementing their own stats kinds would be
aware anyway because we are going to need a callback to initialize the
file to write if we do that, and the redo LSN should be provided in
input of it. Giving more control to extension developers here would
be OK for me, especially since they could use their own format for
their output file(s).
--
Michael