On Fri, Mar 11, 2016 at 7:08 AM, Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Thu, Mar 10, 2016 at 8:26 PM, Alexander Korotkov <
> a(dot)korotkov(at)postgrespro(dot)ru> wrote:
>
>> I don't think we can rely on median that much if we have only 3 runs.
>> For 3 runs we can only apply Kornfeld method which claims that confidence
>> interval should be between lower and upper values.
>> Since confidence intervals for master and patched versions are
>> overlapping we can't conclude that expected TPS numbers are different.
>> Dilip, could you do more runs? 10, for example. Using such statistics we
>> would be able to conclude something.
>>
>
> Here is the reading for 10 runs....
>
>
> Median Result
> -----------------------
>
> Client Base Patch
> -------------------------------------------
> 1 19873 19739
> 2 38658 38276
> 4 68812 62075
>
> Full Results of 10 runs...
>
> Base
> -------------
> Runs 1 Client 2 Client 4 Client
> -----------------------------------------------------
> 1 19442 34866 49023
> 2 19605 35139 51695
> 3 19726 37104 53899
> 4 19835 38439 55708
> 5 19866 38638 67473
> 6 19880 38679 70152
> 7 19973 38720 70920
> 8 20048 38737 71342
> 9 20057 38815 71403
> 10 20344 41423 77953
> -----------------------------------------------------
>
>
> Patch
> -----------
> Runs 1 Client 2 Client 4 Client
> ------------------------------------------------------
> 1 18881 30161 54928
> 2 19415 32031 59741
> 3 19564 35022 61060
> 4 19627 36712 61839
> 5 19670 37659 62011
> 6 19808 38894 62139
> 7 19857 39081 62983
> 8 19910 39923 75358
> 9 20169 39937 77481
> 10 20181 40003 78462
> ------------------------------------------------------
>
I've drawn graphs for these measurements. The variation doesn't look random
here. TPS is going higher from measurement to measurement. I bet you did
measurements sequentially.
I think we should do more measurements until TPS stop growing and beyond to
see accumulate average statistics. Could you please do the same tests but
50 runs.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
