Re: checkpointer continuous flushing

Hello Amit,

> I have tried your scripts and found some problem while using avg.py
> script.
> grep 'progress:' test_medium4_FW_off.out | cut -d' ' -f4 | ./avg.py
> --limit=10 --length=300
> : No such file or directory

> I didn't get chance to poke into avg.py script (the command without
> avg.py works fine). Python version on the m/c, I planned to test is
> Python 2.7.5.

Strange... What does "/usr/bin/env python" say? Can the script be started
on its own at all? I think that the script should work both with python2
and python3, at least it does on my laptop...

> Today while reading the first patch (checkpoint-continuous-flush-10-a),
> I have given some thought to below part of patch which I would like
> to share with you.
>
> + * Select a tablespace depending on the current overall progress.
> + *
> + * The progress ratio of each unfinished tablespace is compared to
> + * the overall progress ratio to find one with is not in advance
> + * (i.e. overall ratio > tablespace ratio,
> + * i.e. tablespace written/to_write > overall written/to_write

> Here, I think above calculation can go for toss if backend or bgwriter
> starts writing buffers when checkpoint is in progress. The tablespace
> written parameter won't be able to consider the one's written by backends
> or bgwriter.

Sure... This is *already* the case with the current checkpointer, the
schedule is performed with respect to the initial number of buffers it
think it will have to write, and if someone else writes these buffers then
the schedule is skewed a little bit, or more... I have not changed this
logic, but I extended it to handle several tablespaces.

If this (the checkpointer progress evaluation used for its schedule is
sometimes wrong because of other writes) is proven to be a major
performance issue, then the processes which writes the checkpointed
buffers behind its back should tell the checkpointer about it, probably
with some shared data structure, so that the checkpointer can adapt its
schedule.

This is an independent issue, that may be worth to address some day. My
opinion is that when the bgwriter or backends quick in to write buffers,
they are basically generating random I/Os on HDD and killing tps and
latency, so it is a very bad time anyway, thus I'm not sure that this is
the next problem to address to improve pg performance and responsiveness.

> Now it may not big thing to worry but I find Heikki's version worth
> considering, he has not changed the overall idea of this patch, but the
> calculations are somewhat simpler and hence less chance of going wrong.

I do not think that Heikki version worked wrt to balancing writes over
tablespaces, and I'm not sure it worked at all. However I reused some of
his ideas to simplify and improve the code.

--
Fabien.