Re: long-standing data loss bug in initial sync of logical replication

On Mon, 2 Sept 2024 at 10:12, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Fri, Aug 30, 2024 at 3:06 PM Shlok Kyal <shlok(dot)kyal(dot)oss(at)gmail(dot)com> wrote:
> >
> > Next I am planning to test solely on the logical decoding side and
> > will share the results.
> >
>
> Thanks, the next set of proposed tests makes sense to me. It will also
> be useful to generate some worst-case scenarios where the number of
> invalidations is more to see the distribution cost in such cases. For
> example, Truncate/Drop a table with 100 or 1000 partitions.
>
> --
> With Regards,
> Amit Kapila.

Also, I did testing with a table with partitions. To test for the
scenario where the number of invalidations are more than distribution.
Following is the test case:
1. Created a publisher on a single table, say 'tconc_1';
2. Created a second publisher on a partition table say 'tp';
3. Created 'tcount' partitions for the table 'tp'.
4. two sessions are running in parallel, let's say S1 and S2.
5. Begin a transaction in S1.
6. S1: Insert a row in table 'tconc_1'
S2: BEGIN; TRUNCATE TABLE tp; COMMIT;
With patch, this will add 'tcount * 3' invalidation messages to
transaction in session 1.
S1: Insert a row in table 't_conc1'
7. COMMIT the transaction in S1.
8. run 'pg_logical_slot_get_binary_changes' to get the decoding changes.

Performance:
We see a degradation in performance. Results are an average of 5 runs.

count of partitions | Head (sec) | Fix (sec) | Degradation (%)
-------------------------------------------------------------------------------------
1000 | 0.114 | 0.118 | 3.50877193
5000 | 0.502 | 0.522 | 3.984063745
10000 | 1.012 | 1.024 | 1.185770751

I have also attached the test script here. And will also do further testing.

Thanks and Regards,
Shlok Kyal