Re: checkpointer continuous flushing

Hello Amit,

>> medium2: scale=300 shared_buffers=5GB checkpoint_timeout=30min
>> max_wal_size=4GB warmup=1200 time=7500
>>
>> flsh | full speed tps | percent of late tx, 4 clients
>> /srt | 1 client | 4 clients | 100 | 200 | 400 |
>> N/N | 173 +- 289* | 198 +- 531* | 27.61 | 43.92 | 61.16 |
>> N/Y | 458 +- 327* | 743 +- 920* | 7.05 | 14.24 | 24.07 |
>> Y/N | 169 +- 166* | 187 +- 302* | 4.01 | 39.84 | 65.70 |
>> Y/Y | 546 +- 143 | 681 +- 459 | 1.55 | 3.51 | 2.84 |
>>
>> The effect of sorting is very positive (+150% to 270% tps). On this run,
> flushing has a positive (+20% with 1 client) or negative (-8 % with 4
> clients) on throughput, and late transactions are reduced by 92-95% when
> both options are activated.
>
> Why there is dip in performance with multiple clients,

I'm not sure to see the "dip". The performances are better with 4 clients
compared to 1 client?

> can it be due to reason that we started doing more stuff after holding
> bufhdr lock in below code?

I think it is very unlikely that the buffer being locked would be
simultaneously requested by one of the 4 clients for an UPDATE, so I do
not think it should have a significant impact.

> BufferSync() [...]

> BufferSync()
> {
> ..
> - buf_id = StrategySyncStart(NULL, NULL);
> - num_to_scan = NBuffers;
> + active_spaces = nb_spaces;
> + space = 0;
> num_written = 0;
> - while (num_to_scan-- > 0)
> +
> + while (active_spaces != 0)
> ..
> }
>
> The changed code doesn't seems to give any consideration to
> clock-sweep point

Indeed.

> which might not be helpful for cases when checkpoint could have flushed
> soon-to-be-recycled buffers. I think flushing the sorted buffers w.r.t
> tablespaces is a good idea, but not giving any preference to clock-sweep
> point seems to me that we would loose in some cases by this new change.

I do not see how to do both, as these two orders seem more or less
unrelated? The traditionnal assumption is that the I/O are very slow and
they are to be optimized first, so going for buffer ordering to be nice to
the disk looks like the priority.

--
Fabien.