Re: Reliable WAL file shipping over unreliable network

> On Mar 5, 2018, at 10:02 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
>
> It doesn’t- but please don't encourage partial solutions which have ver clear issues.

Then problem is there are no good base utilities that is useful with archive_command; unless you’re writing directly to an NFS mount or a tape library. Even Barman recommends rsync with the archive_command; if you are unable to use pg_receivexlog solution. There are countless Postgres documentation out there that recommends use of rsync with the archive_command.

Here a solution that will fsync() file on the other end.

SSH_CMD="ssh -o ServerAliveInterval=20 $ARCH_SERVER"
STS=3

OUTPUT=$(cat $XLOGFILE | $SSH_CMD "(mkdir -p $ARCH_DIR && ~/bin/fwrite $ARCH_DIR/$WALFILE)")
if [ $? == 0 ]; then
STS=0
fi

exit $STS

fwrite code:

#include <sys/stat.h>
#include <err.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define BUFSIZE 131072

int
main(int argc, char *argv[])
{
int fd, r, w;
char *buf;
char *name;

if (argc != 2) {
fprintf(stderr, "usage: fwrite [file]\n");
exit(1);
}

if ((buf = malloc(BUFSIZE)) == NULL)
err(1, "malloc");

++argv;
if ((name = (char *) malloc(strlen(*argv) + 10)) == NULL)
err(1, "malloc");

strcat(strcpy(name, *argv), ".XXXXXX");

if ((fd = mkstemp(name)) < 0)
err(1, "mkstemp");

while ((r = read(STDIN_FILENO, buf, BUFSIZE)) > 0)
if ((w = write(fd, buf, r)) == -1)
err(1, "write");

if (r < 0)
err(1, "read");

if (fsync(fd) != 0)
err(1, "fsync");

if (close(fd) != 0)
err(1, "close");

if (rename(name, *argv) != 0)
err(1, "rename");

free(name);
exit(0);
}