Re: [PATCH] pg_basebackup: progress report max once per second

On Fri, Jan 10, 2014 at 3:56 AM, Mika Eloranta <mel(at)ohmu(dot)fi> wrote:

> On Thu, Jan 9, 2014, at 20:05, Magnus Hagander wrote:
> > On Thu, Nov 14, 2013 at 10:27 AM, Mika Eloranta <mel(at)ohmu(dot)fi> wrote:
> >> On 13 Nov 2013, at 20:51, Mika Eloranta <mel(at)ohmu(dot)fi> wrote:
> >>
> >>
> >>
> > Prevent excessive progress reporting that can grow to gigabytes
> >>
> > of output with large databases.
> >>
> >>
> >> Same patch as an attachment.
> >
> > Would it not make more sense to instead store the last number printed,
> and only print it if the percentage has changed? AIUI with this patch we
> still print the same thing on top of itself if it takes >1 second to get 1%
> further.
> >
> > (Except for verbose mode - but if you're asking for verbose mode, you
> are *asking* to get lots of output)
>
> (re-sent response as I used the wrong sender address previously, sorry
> about the dupe)
>
> Printing out progress periodically is IMHO slightly better as the
> interactive user can see that there is some progress (e.g. by pressing
> enter for a new empty console line) during a huge basebackup operation.
>

That's an argument I hadn't considered - but I still think it's acceptable
to wait until the next percentage digit in this case.

The original problem I wanted to address was that I had a daemon
> watching over the basebackup process and reading its output to make sure
> that the basebackup is proceeding properly. It also wrote all the output
> to a logfile for postmortem analysis. The log file grew to a very big
> size (multiple gigabytes due to the progress prints). With my patch the
> log was only a few kilos, without any negative effects. The excessive
> progress reporting can also be an issue in an interactive session over a
> slow network (mobile), hogging both time and bandwidth.
>

Yeah, I was guessing it was something like that. I didn't realize you'd
actually monitor it through a deamon (I've just looked at the output
filesize when minitoring things like that), but the remote connection was
easily reproducible. I definitely agree we should do something about it.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/