RE: long-standing data loss bug in initial sync of logical replication

On Friday, August 9, 2024 7:21 PM Shlok Kyal <shlok(dot)kyal(dot)oss(at)gmail(dot)com> wrote:

Hi,

>
> In the v7 patch, I am looping through the reorder buffer of the current committed
> transaction and storing all invalidation messages in a list. Then I am
> distributing those invalidations.
> But I found that for a transaction we already store all the invalidation messages
> (see [1]). So we don't need to loop through the reorder buffer and store the
> invalidations.
>
> I have modified the patch accordingly and attached the same.

I have tested this patch across various scenarios and did not find issues.

I confirmed that changes are correctly replicated after adding the table or
schema to the publication, and changes will not be replicated after removing
the table or schema from the publication. This behavior is consistent in both
streaming and non-streaming modes. Additionally, I verified that invalidations
occurring within subtransactions are appropriately distributed.

Please refer to the attached ISOLATION tests which tested the above cases.
This also inspires me if it would be cheaper to write an ISOLATION test for this
bug instead of building a real pub/sub cluster. But I am not against the current
tests in the V8 patch as that can check the replicated data in a visible way.

Best Regards,
Hou zj