Re: [(catch-ext)] Re: [(catch-ext)] Re: Setting up streaming replication on large database (20+ TB) for the first time

Yes, pgbackrest seems the best solution for faster backup and restore.

We can close the discussion here for now, replaying million WAL files is
just slow and this is normal.

On Thu, Aug 18, 2022 at 12:10 PM Vijaykumar Jain <
vijaykumarjain(dot)github(at)gmail(dot)com> wrote:

> Sorry for top posting, from phone.
>
> But pgbackrest exactly helped with that. With compression and parallel
> process in backup, the backup and restore was quick. I used this, where I
> took a backup and immediately did a restore so less wals to replay, else
> wal replay is indeed slow.
>
> On Thu, Aug 18, 2022, 1:03 PM Ivan N. Ivanov <ivan(dot)ni(at)telebid-pro(dot)com>
> wrote:
>
>> Thank you, people. The big problem in my case, which I have not
>> mentioned, is that I think the network is a bottleneck, because I am
>> running pg_basebackup through internet from local country to Amazon
>> instance in Germany and the speed in copying is around 50 MB/sec max, that
>> is why it takes 2 days for copying.
>>
>> I will try using high compression for the basebackup to reduce the time.
>>
>> pgbackrest is an alternative, too
>>
>> Thank you again!
>>
>>
>>
>> On Wed, Aug 17, 2022 at 11:13 PM Ivan N. Ivanov <ivan(dot)ni(at)telebid-pro(dot)com>
>> wrote:
>>
>>> Thank you for your answer! I have found this tool and I will try it
>>> tomorrow to see if this "read-ahead" feature will speed up the process.
>>>
>>> On Wed, Aug 17, 2022 at 11:09 PM Christophe Pettus <xof(at)thebuild(dot)com>
>>> wrote:
>>>
>>>>
>>>>
>>>> > On Aug 17, 2022, at 13:06, Ivan N. Ivanov <ivan(dot)ni(at)telebid-pro(dot)com>
>>>> wrote:
>>>> >
>>>> > How to speed up recovering of WAL files?
>>>>
>>>> Since you are running on your own hardware, you might take a look at:
>>>>
>>>> https://github.com/TritonDataCenter/pg_prefaulter
>>>>
>>>