Re: checkpointer continuous flushing

Hello Andres,

> [...] Right. At the cost of keeping track of all files...

Sure. Pg already tracks all files, and probably some more tracking would
be necessary for an early fsync feature to know what are those already
fsync'ed and what are those not yet fsync'ed.

> Yes, the other version has a higher space overhead.

Yep, this is my concern.

> I'm not convinced that's meaningful in comparison to shared buffers in
> space. And rather doubtful it a loss performance wise in a loaded
> server. All the buffer headers are touched on other cores and doing the
> sort with indirection will greatly increase bus traffic.

The measures I collected and reported showed that the sorting time is
basically insignificant, so bus traffic induced by sorting does not seem
to be an issue.

> [...] It's not particularly desirable to have a performance feature that
> works less well if the server is heavily and concurrently loaded. The
> likelihood of bogus sort results will increase with the churn rate in
> shared buffers.

Hm.

In conclusion I'm not convinced that it is worth the memory, but I'm also
tired of arguing, and hopefully nobody else cares about a few more bytes
per shared_buffers, so why should I care?

Here is a v8, I reduced the memory overhead of the "heavy weight" approach
from 24 to 16 bytes per buffer, so it is medium weight:-). It might be
compacted further down to 12 bytes by combining the 2 bits of forkNum
either with relNode or blockNum, and use a uint64_t comparison field with
all data so that the comparison code would be simpler and faster.
I also fixed the computation of the shmem size which I had not updated
when switching to shmem.

The patches still include the two guc, but it is easy to remove one or the
other. They are useful is someone wants to test. The default is on for
sort, and off for flush. Maybe it should be on for both.

--
Fabien.