Re: gitlab post-mortem: pg_basebackup waiting for checkpoint

On 02/17/2017 08:17 PM, Jim Nasby wrote:
> On 2/14/17 5:18 PM, Robert Haas wrote:
>> On Tue, Feb 14, 2017 at 4:06 PM, Alvaro Herrera
>> <alvherre(at)2ndquadrant(dot)com> wrote:
>>> I'd rather have a --quiet mode instead. If you're running it by hand,
>>> you're likely to omit the switch, whereas when writing the cron job
>>> you're going to notice lack of switch even before you let the job run
>>> once.
>>
>> Well, that might've been a better way to design it, but changing it
>> now would break backward compatibility and I'm not really sure that's
>
> Meh... it's really only going to affect cronjobs or scripts, which are
> easy enough to fix, and you're not going to have that many of them (or
> if you do you certainly have an automated way to push the update).
>

I think you're underestimating the breakage and overestimating how easy
it's going to be to it. It's true we'd only change this in a major
version, so people should assume possible breakage and test.

>> a good idea. Even if it is, it's a separate concern from whether or
>> not in the less-quiet mode we should point out that we're waiting for
>> a checkpoint on the server side.
>
> Well, --quite was suggested because of confusion from pg_basebackup
> twiddling it's thumbs...

I'm in favor of the '--verbose' route. People are used to that when
investigating issues, and it does not break existing cron jobs. I can
live with --quiet though, as long as we don't resort to some craziness
along the lines "if there's tty be verbose, otherwise be quiet".

I have my doubts about this actually addressing gitlab-like mistakes,
though, because it's a helluva jump from "It's waiting and not doing
anything," to "We need to remove the datadir." (One of the reasons being
that non-empty directory is a local issue, and there's no reason why the
tool should wait instead of just reporting an error.)

FWIW before messing with the pg_basebackup code, perhaps we should
improve the documentation and explain clearly the meaning of 'fast' and
'spread' checkpoint modes. Right now, pg_basebackup docs only say this:

Sets checkpoint mode to fast or spread (default) (see Section 24.3.3).

which is pretty damn useless, when you're investigating an issue. And
the referenced section (Making a Base Backup Using the Low Level API)
does not clearly explain how this maps to pg_start_backup(_,?).

What about adding a paragraph into pg_basebackup docs, explaining that
with 'fast' it does immediate checkpoint, while with 'spread' it'll wait
for a spread checkpoint.

regards

-- Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services