Re: refactoring basebackup.c

On Wed, May 13, 2020 at 12:01 AM Suraj Kharage <
suraj(dot)kharage(at)enterprisedb(dot)com> wrote:

> 8kb 32kb (default value) 128kB 1024kB
> Without refactor patch real 10m22.718s
> user 1m23.629s
> sys 8m51.410s real 8m36.245s
> user 1m8.471s
> sys 7m21.520s real 6m54.299s
> user 0m55.690s
> sys 5m46.502s real 18m3.511s
> user 1m38.197s
> sys 9m36.517s
> With refactor patch (Robert's patch) real 10m11.350s
> user 1m25.038s
> sys 8m39.226s real 8m56.226s
> user 1m9.774s
> sys 7m41.032s real 7m26.678s
> user 0m54.833s
> sys 6m20.057s real 18m17.230s
> user 1m42.749s
> sys 9m53.704s
>
> The above numbers are taken from the minimum of two runs of each scenario.
>
> I can see, when we have TAR_SEND_SIZE as 32kb or 128kb, it is giving us a
> good performance whereas, for 1Mb it is taking 2.5x more time.
>
> Please let me know your thoughts/suggestions on the same.
>

So the patch came out slightly faster at 8kB and slightly slower in the
other tests. That's kinda strange. I wonder if it's just noise. How much do
the results vary run to run?

I would've expected (and I think Andres thought the same) that a smaller
block size would be bad for the patch and a larger block size would be
good, but that's not what these numbers show.

I wouldn't worry too much about the regression at 1MB. Probably what's
happening there is that we're losing some concurrency - perhaps with
smaller block sizes the OS can buffer the entire chunk in the pipe
connecting pg_basebackup to the server and start on the next one, but when
you go up to 1MB it doesn't fit and ends up spending a lot of time waiting
for data to be read from the pipe. Wait event profiling might tell you
more. Probably what this suggests is that you want the largest buffer size
that doesn't cause you to overrun the network/pipe buffer and no larger.
Unfortunately, I have no idea how we'd figure that out dynamically, and I
don't see a reason to believe that everyone will have the same size buffers.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company