Re: Review: Revise parallel pg_restore's scheduling heuristic

On Mon, Aug 03, 2009 at 10:03:47AM -0500, Kevin Grittner wrote:
> That's about 0.52% slower with the patch. Because there was over 10%
> variation in the numbers with the patch, I tried leaving out the four
> highest outliers on both, in case it was the result of some other
> activity on the system (even though this machine should have been
> pretty quiet over the weekend) and the difference fell to 0.09%.

What do people do when testing this? I think I'd look to something like
Student's t-test to check for statistical significance. My working
would go something like:

I assume the variance is the same because it's being tested on the
same machine.

samples = 20
stddev = 144.26
avg1 = 4783.13
avg2 = 4758.46
t = 0.54 ((avg1 - avg2) / (stddev * sqrt(2/samples)))

We then have to choose how certain we want to be that they're actually
different, 90% is a reasonably easy level to hit (i.e. one part in ten,
with 95% being more commonly quoted). For 20 samples we have 19 degrees
of freedom--giving us a cut-off[1] of 1.328. 0.54 is obviously well
below this allowing us to say that there's no "statistical significance"
between the two samples at a 90% level.

--
Sam http://samason.me.uk/

[1] http://en.wikipedia.org/wiki/Student's_t-distribution#Table_of_selected_values