Re: [patch] pg_copy - a command for reliable WAL archiving

From: "Fujii Masao" <masao(dot)fujii(at)gmail(dot)com>
> What's the main purpose of this tool? If it's for WAL archiving, the tool
> name
> "pg_copy" sounds too generic. We already have pg_archivecleanup, so maybe
> "pg_archivecopy" or something is better for the consistency?
>
> pg_copy in the patch copies the file to the destination in a
> straightforward way,
> i.e., directly copies the file to the dest file with actual name. This can
> cause
> the problem which some people reported. The problem is that, when the
> server
> crashes while WAL file is being archived by cp command, its
> partially-filled
> WAL file remains at the archival area. This half-baked archive file can
> cause
> various troubles. To address this, WAL file needs to be copied to the
> temporary
> file at first, then renamed to the actual name. I think that pg_copy
> should
> copy the WAL file in that way.

I intended to make pg_copy a straightforward replacement of cp/copy, which
complements the missing sync. Direct I/O and posix_fadvice() feature may be
convenient but not essential for this utility. cp/copy doesn't copy to a
temporary file, and the problem can be solved easily by mv/move. I wanted
to keep pg_copy as generic as cp/copy, so that it can be used by some
advanced features in the future, e.g. comprehensive backup/recovery
management like RMAN (this example may not be best) when it's integrated
into the core.

With that said, copying to a temporary file like <dest>.tmp and renaming it
to <dest> sounds worthwhile even as a basic copy utility. I want to avoid
copying to a temporary file with a fixed name like _copy.tmp, because some
advanced utility may want to run multiple instances of pg_copy to copy
several files into the same directory simultaneously. However, I'm afraid
multiple <dest>.tmp files might continue to occupy disk space after
canceling copy or power failure in some use cases, where the copy of the
same file won't be retried. That's also the reason why I chose to not use a
temporary file like cp/copy.

> Currently pg_copy always syncs the archive file, and there is no way to
> disable
> that. But I'm sure that not everyone want to sync the archive file. So I
> think
> that it's better to add the option specifying whether to sync the file
> or not, into
> pg_copy.

pg_copy is for copying a file reliably by syncing. If sync is not
necessary, people can use cp/copy.

> Some users might want to specify whether to call posix_fadvise or not
> because
> they might need to re-read the archvied files just after the archiving.
> For example, network copy of the archived files from the archive area to
> remote site for disaster recovery.

This sounds reasonable. Do you have an idea on the switch name and the
default behavior?

> Do you recommend to use pg_copy for restore_command? If yes, it also
> should
> be documented. And in the WAL restore case, the restored WAL files are
> re-read
> soon by recovery, so posix_fadvise is not good in that case.
>
> Direct I/O and posix_fadvise are used only for destination file. But why
> not
> source file? That might be useful especially for restore_command case.

No, I don't think it's necessary to use pg_copy for restore_command.

> At last, the big question is, is there really no OS command which provides
> the same functionality as pg_copy does? If there is, I'd like to avoid
> duplicate
> work basically.

If there exists such a command available in the standard OS installation, I
want to use it.

Regards
MauMau