Re: backup tools ought to ensure created backups are durable

On Mon, Mar 28, 2016 at 3:12 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:

> On 2016-03-28 11:35:57 +0200, Magnus Hagander wrote:
> > On Mon, Mar 28, 2016 at 3:11 AM, Michael Paquier <
> michael(dot)paquier(at)gmail(dot)com>
> > wrote:
> >
> > > On Mon, Mar 28, 2016 at 8:30 AM, Andres Freund <andres(at)anarazel(dot)de>
> wrote:
> > > > As pointed out in
> > > >
> > >
> http://www.postgresql.org/message-id/20160327232509.v5wgac5vskusedin@awork2.anarazel.de
> > > > our backup tools (i.e. pg_basebackup, pg_dump[all]), currently don't
> > > > make any efforts to ensure their output is durable.
> > > >
> > > > I think for backup tools of possibly critical data, that's pretty
> much
> > > > unaceptable.
> > >
> > > Definitely agreed, once a backup/dump has been taken and those
> > > utilities exit, we had better ensure that they are durably on disk.
> > > For pg_basebackup and pg_dump, as everything except pg_dump/plain
> > > require a target directory for the location of the output result, we
> > > really can make things better.
> > >
> > >
> > Definitely agreed on fixing it. But I don't think your summary is right.
> >
> > pg_basebackup in tar mode can be sent to stdout, does not require a
> > directory. And the same for pg_dump in any mode except for directory. So
> we
> > can't just drive it off the mode, some more detailed checks are required.
>
> if (!isastty(stdout)) ought to do the trick, afaics? And maybe add a
> warning somewhere in the docs about the tools not fsyncing if you pipe
> their output data somewhere?
>

That should work yeah. And given that we already use that check in other
places, it seems it should be perfectly safe. And as long as we only do a
WARNING and not abort if the fsync fails, we should be OK if people
intentionally store their backups on an fs that doesn't speak fsync (if
that exists), in which case I don't really think we even need a switch to
turn it off.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/