Re: Load distributed checkpoint V4

On Thu, 19 Apr 2007, Heikki Linnakangas wrote:

> In the sync phase, we sleep between each fsync until enough time/segments
> have passed, assuming that the time to fsync is proportional to the file
> length. I'm not sure that's a very good assumption.

I've been making scatter plots of fsync time vs. amount written to the
database for a couple of months now, and while there's a trend there it's
not a linear one based on data written. Under Linux, to make a useful
prediction about how long a fsync will take you first need to consider how
much dirty data is already in the OS cache (the "Dirty:" figure in
/proc/meminfo) before the write begins, relative to the kernel parameters
that control write behavior. Combine that with some knowledge of the
caching behavior of the controller/disk combination you're using, and it's
just barely possible to make a reasonable estimate. Any less information
than all that and you really have very little basis on which to guess how
long it's going to take.

Other operating systems are going to give completely different behavior
here, which of course makes the problem even worse.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD