Re: [RFC] What should we do for reliable WAL archiving?

From: "Mitsumasa KONDO" <kondo(dot)mitsumasa(at)gmail(dot)com>
> 2014-03-17 21:12 GMT+09:00 Fujii Masao <masao(dot)fujii(at)gmail(dot)com>:
>
>> On Mon, Mar 17, 2014 at 10:20 AM, Robert Haas <robertmhaas(at)gmail(dot)com>
>> wrote:
>> > On Sun, Mar 16, 2014 at 6:23 AM, MauMau <maumau307(at)gmail(dot)com> wrote:
>> >> * Improve the example in the documentation.
>> >> But what command can we use to reliably sync just one file?
>> >>
>> >> * Provide some command, say pg_copy, which copies a file synchronously
>> by
>> >> using fsync(), and describes in the doc something like "for simple use
>> >> cases, you can use pg_copy as the standard reliable copy command."
>> >
>> > +1. This won't obviate the need for tools to manage replication, but
>> > it would make it possible to get the simplest case right without
>> > guessing.
>>
>> +1, too.
>>
>> And, what about making pg_copy call posix_fadvise(DONT_NEED) against the
>> archived file after the copy? Also It might be good idea to support the
>> direct
>> copy of the file to avoid wasting the file cache.
>
> Use direct_cp.
> http://directcp.sourceforge.net/direct_cp.html

Thank you all for giving favorable responses and interesting ideas.
Then, I think I'll do:

* Create pg_copy in C so that it can be used on Windows as well as on
UNIX/Linux. It just copies one file. Its source code is located in
src/bin/pg_copy/. Please recommend a better name if you have one in mind.

* Add a reference page for pg_copy in the chapter "Server applications".
Modify the section for continuous archiving to recommend pg_copy for simple
use cases as the standard command.

* pg_copy calls posix_fadvise(DONT_NEED) on the destination file.

* pg_copy passes O_DIRECT flag when opening the destination file
when --directio or -d option is specified. O_DIRECT is not used by default
because it may not be available on some file systems, as well as it might
cause trouble on older platforms such as RHEL4/5. pg_copy does not use
O_DIRECT for the source file so that it can copy the data from the
filesystem cache, which is just written by postgres.

Could you give me your opinions before starting the work, including the
following?

* Should I refactor the functions (copy_file, copydir, etc.) in
src/backend/storage/file/copydir.c so that they can also be used for
frontends? If so, which of src/port or src/common/ is the right place to
put copydir.c in?

* Should I complete the work before 9.4 beta so that it will be available
starting with 9.4? I think so because it is a basic capability to archive
transaction logs safely (although the time may not allow me to do this).

Regards
MauMau