Re: Reliable WAL file shipping over unreliable network

Just use "-ac”; you want -c option to ensure no data corruption during the transfer. Do not delete the file; let Postgres manage that.

Here is a snippet from I script I use for archiving. You also want to make your script returns failure or success correctly.

# SSH Command and options
SSH_CMD="ssh -o ServerAliveInterval=20 $ARCH_SERVER"
STS=3

OUTPUT=$(rsync -ac --rsync-path="mkdir -p $ARCH_DIR && rsync" $XLOGFILE $ARCH_SERVER:$ARCH_DIR/$WALFILE)
if [ $? == 0 ]; then
STS=0
fi

exit $STS

> On Feb 28, 2018, at 11:38 AM, scott ribe <scott_ribe(at)elevated-dev(dot)com> wrote:
>
> On Feb 28, 2018, at 8:46 AM, Nagy László Zsolt <gandalf(at)shopzeus(dot)com> wrote:
>>
>> Another problem with rsync might be that it does not know if a WAL file
>> on the source side is growing (e.g. being written) and it might start
>> copy that before it is fully flushed to disk. I see this as a big
>> problem, but I don't have experience.
>
> You use rsync in the archive command, not by itself. Postgres does not try to copy the file until it is ready.
>
>> E.g. move the file into the archive dir only after it has been fully
>> copied to the destination volume. But then we still have the problem of
>> partially written files on the slave side.
>
> No, you do not have that problem at all.
>
> Given an unreliable network, you probably do want to use --partial
>
>
> --
> Scott Ribe
> https://www.linkedin.com/in/scottribe/
> (303) 722-0567
>
>