Re: Parallelize stream replication process

> On Sep 16, 2020, at 11:15 AM, Li Japin <japinli(at)hotmail(dot)com> wrote:
>
>
>
>> On Sep 15, 2020, at 3:41 PM, Fujii Masao <masao(dot)fujii(at)oss(dot)nttdata(dot)com> wrote:
>>
>>
>>
>> On 2020/09/15 13:41, Bharath Rupireddy wrote:
>>> On Tue, Sep 15, 2020 at 9:27 AM Li Japin <japinli(at)hotmail(dot)com> wrote:
>>>>
>>>> For now, postgres use single process to send, receive and replay the WAL when we use stream replication,
>>>> is there any point to parallelize this process? If it does, how do we start?
>>>>
>>>> Any thoughts?
>>
>> Probably this is another parallelism than what you're thinking,
>> but I was thinking to start up walwriter process in the standby server
>> and make it fsync the streamed WAL data. This means that we leave
>> a part of tasks of walreceiver process to walwriter. Walreceiver
>> performs WAL receive and write, and walwriter performs WAL flush,
>> in parallel. I'm just expecting that this change would improve
>> the replication performance, e.g., reduce the time to wait for
>> sync replication.

Yes this should be able to improve that in theory.

>>
>> Without this change (i.e., originally), only walreceiver performs
>> WAL receive, write and flush. So wihle walreceiver is fsyncing WAL data,
>> it cannot receive newly-arrived WAL data. If WAL flush takes a time,
>> which means that the time to wait for sync replication in the primary
>> would be enlarged.
>>
>> Regards,
>>
>> --
>> Fujii Masao
>> Advanced Computing Technology Center
>> Research and Development Headquarters
>> NTT DATA CORPORATION
>
> Yeah, this might be a direction.
>
> Now I am thinking about how to parallel WAL log replay. If we can improve the efficiency
> of replay, then we can shorten the database recovery time (streaming replication or database
> crash recovery).

Yes, parallelization should be able to help the scenario that cpu util rate is 100% or if it is not
100% but continues blocking in some operations which could be improved by running in
parallel. There are a lot of scenarios of WAL replay (memory operation, system calls, locking, etc).
I do not have the experience of real production environment, so not sure whether or how
the recovery suffer, but I believe parallel recovery should help to accelerate and help failover
which is quite important especially in cloud environment. To do this may need to carefully
analyze the dependency of various WALL at first. It won’t be a small effort. I’ve heard some
databases have implemented this though not sure how much that helps.

For parallel wal receiver/sender, its functionality is simple so after decoupling the fsync operation to
a separate process not sure how beneficial parallelization would be (surely if wal receiver/sender
Is 100% cpu utilized that is needed).

>
> For streaming replication, we may need to improve the transmission of WAL logs to improve
> the entire recovery process.
>
> I’m not sure if this is correct.
>
> Regards,
> Japin Li.
>