RE: pg_basebackup inconsistent performance

I'm guessing the above happens from one run to another correct?
Yes

Where are the machines you are backing up from/to relative to each on the network?
Direct ethernet connection between 10G network interfaces

Is there increased activity on the database servers e.g. inserts, updates, etc during the extended backups?
Not according to sar reports

Jasen M. Lentz, M.Ed
Lead Systems Administrator
Sesco Enterprises, LLC
4977 State Route 30 East (Mailing Address Only)
Greensburg, PA 15601
W: (724) 837-1991 x207
C: (412) 848-5612

-----Original Message-----
From: Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com>
Sent: Wednesday, May 6, 2020 10:28 AM
To: Jasen Lentz <jlentz(at)sescollc(dot)com>; pgsql-general(at)lists(dot)postgresql(dot)org
Subject: Re: pg_basebackup inconsistent performance

On 5/6/20 5:44 AM, Jasen Lentz wrote:
> We have pg_basebackup running on two of our DB servers that are
> replicated.  We are running postgres 11, and it seems we are getting
> inconsistent performance from the backups and unsure as of why.  We
> start out at 5-6 hours over a dedicated 10G port for 7TB.  It creeps
> up to 8-9 hours then all of a sudden takes 12-16 hours.  There seems
> to be

I'm guessing the above happens from one run to another correct?

Where are the machines you are backing up from/to relative to each on the network?

Is there increased activity on the database servers e.g. inserts, updates, etc during the extended backups?

> no rhyme or reason for the extended backup times.  The command we use
> for backups is as follows:
>
> On server 2 (secondary), starts at 4PM
>
> pg_basebackup --pgdata=/opt/postgres/pgbackup/`echo $DATE`
> --format=plain --write-recovery-conf --no-sync --wal-method=stream
> --checkpoint=fast --label=`hostname`-`echo $DATE`
> --no-verify-checksums --host=<server1> --username=replication
> --port=5432
>
> On server 1 (Primary), starts at Midnight
>
> pg_basebackup --pgdata=/opt/postgres/pgbackup/`echo $DATE`
> --format=plain --write-recovery-conf --no-sync --wal-method=stream
> --checkpoint=fast --label=`hostname`-`echo $DATE`
> --no-verify-checksums --host=<server2> --username=replication
> --port=5432
>
> I'm not sure why or how we are running into the weeds.  I am the
> SysAdmin and am not familiar with the inner workings of the DB.  I can
> pass any commands that need run along to our DBA.
>
> From the OS perspective, we are not seeing any problems with CPU,
> memory or disk.  We are running on RHEL 7.7
>
> Thanks!
>

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com