Re: Parallel WAL Archival Options

Hi Ron,

Uploads to a *remote* server?
— Yes, to S3

Does wal-g compress files before sending them across the wire? By how
much? Are you CPU or IO bound by having to compress that much data?
— Yes, we use default compression i.e lz4. There is no pressure on
resources but the number of wal files that are processed in parallel, if we
increase the streams it uses up a lot of memory.

Thanks,
Nikhil

On Sun, 6 Aug 2023 at 13:51, Ron <ronljohnsonjr(at)gmail(dot)com> wrote:

> On 8/6/23 02:43, Nikhil Shetty wrote:
>
> Hi Team,
>
> I would like to know which backup&restore tools will be better for
> scenarios where the database is generating around 400 WALs per minute.
>
>
> If my math is correct, 400x 16MB WAL files per minute is
> 400*(16*2^20)/60*8 / 10^6 = 895 MBits *per second*. Plus overhead.
>
> That's about *1Gbit/second*. Definitely nothing to sneeze at.
>
> We are using wal-g but it is not able to keep pace with the wal
> generation. We increased the upload streams to 256 but no luck
>
>
> Uploads to a *remote* server?
>
> Does wal-g compress files before sending them across the wire? By how
> much? Are you CPU or IO bound by having to compress that much data?
>
>
> --
> Born in Arizona, moved to Babylonia.
>