Re: Backup throttling

On 08/22/2013 03:33 PM, Craig Ringer wrote:
> On 08/22/2013 01:39 PM, PostgreSQL - Hans-Jürgen Schönig wrote:
>
>> what would be a reasonable scenario where limiting streaming would make sense? i cannot think of any to be honest.
>
> I tend to agree. If anything we're likely to want the reverse - the
> ability to throttle WAL *generation* on the master so streaming can keep up.

(I assume you mean WAL *sending* rather than *generation*.)

I don't think we want to throttle WAL sending/receiving at all. The WAL
senders should always keep up with the amount of WAL generated. If the
rate of WAL sending is restricted, then the pg_basebackup (or another
client application) might never catch up.

Also, IMO, pg_basebackup starts at the current WAL segment. Thus the
unthrottled WAL transfer shouldn't cause significant additional load,
such as reading many old WAL segments from the master's disk.

> I see a lot of value in throttling base backup transfer rates. It's
> something PgBarman does per-tablespace using rsync at the moment, but
> it'd be nice if it available as an option possible over the streaming
> replication protocol via pg_basebackup so it was easier for people to
> use ad-hoc and without all the shell access wrangling.

(Possibly huge) DATA directory (tablespaces, etc.) is the actual purpose
of this patch. This is the additional load that pg_basebackup imposes on
the master. As pointed out elsewhere in the thread, the client-side
throttling was chosen because it seemed to be less invasive. But the
discussion (including this your comment) keeps me no longer convinced
that it's the best way. I'll reconsider things.

// Antonin Houska (Tony)