Re: Improvement of checkpoint IO scheduler for stable transaction responses

On Mon, Jun 17, 2013 at 2:18 AM, Andres Freund <andres(at)2ndquadrant(dot)com>wrote:

> On 2013-06-16 17:27:56 +0300, Heikki Linnakangas wrote:
>
> > A long time ago, Itagaki wrote a patch to sort the checkpoint writes:
> www.postgresql.org/message-id/flat/20070614153758(dot)6A62(dot)ITAGAKI(dot)TAKAHIRO(at)oss(dot)ntt(dot)co(dot)jp
> .
> > He posted very promising performance numbers, but it was dropped because
> Tom
> > couldn't reproduce the numbers, and because sorting requires allocating a
> > large array, which has the risk of running out of memory, which would be
> bad
> > when you're trying to checkpoint.
>
> Hm. We could allocate the array early on since the number of buffers
> doesn't change. Sure that would be pessimistic, but that seems fine.
>
> Alternatively I can very well imagine that it would still be beneficial
> to sort the dirty buffers in shared buffers. I.e. scan till we found 50k
> dirty pages, sort them and only then write them out.
>
>
Without knowing that Itagaki had done something similar in the past, couple
of months back I tried exactly the same thing i.e. sort the shared buffers
in chunks and then write them out at once. But I did not get any
significant performance gain except when the shared buffers are 3/4th (or
some such number) or more than the available RAM. I will see if I can pull
out the patch and the numbers. But if memory serves well, I concluded that
the kernel is already utilising its buffer cache to achieve the same thing
and it does not help beyond a point.

Thanks,
Pavan

--
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee