Re: pgBackRest for a 50 TB database

Greetings,

On Thu, Oct 5, 2023 at 03:10 Abhishek Bhola <abhishek(dot)bhola(at)japannext(dot)co(dot)jp>
wrote:

> Here is the update with compress-type=zst in the config file
> Process-max is still 30. *But it longer than before, around 27 hours 50
> mins*
>
> full backup: 20231004-130621F
> timestamp start/stop: 2023-10-04 13:06:21+09 / 2023-10-05
> 15:56:03+09
> wal start/stop: 000000010001AC0E00000054 /
> 000000010001AC0E00000054
> database size: 38249.0GB, database backup size: 38249.0GB
> repo1: backup size: 5799.8GB
>
> Do you think I could be missing something?
>

Sounds like there’s something else which is the bottleneck once you have
process-max at 30. I suspect you could reduce that process-max value and
have around the same time still with zstd. Ultimately if you want it to be
faster then you’ll need to figure out what the bottleneck is (seemingly not
CPU, unlikely to be memory, so that leaves network or storage) and address
that.

We’ve seen numbers approaching 10TB/hr with lots of processes and zstd and
fast storage on high end physical hardware.

Thanks,

Stephen