Re: Column Filtering in Logical Replication

On Fri, Mar 18, 2022 at 12:47 AM Tomas Vondra
<tomas(dot)vondra(at)enterprisedb(dot)com> wrote:
>
> I pushed the second fix. Interestingly enough, wrasse failed in the
> 013_partition test. I don't see how that could be caused by this
> particular commit, though - see the pgsql-committers thread [1].
>

I have a theory about what's going on here. I think this is due to a
test added in your previous commit c91f71b9dc. The newly added test
added hangs in tablesync because there was no apply worker to set the
state to SUBREL_STATE_CATCHUP which blocked tablesync workers from
proceeding.

See below logs from pogona [1].
2022-03-18 01:33:15.190 CET [2551176][client
backend][3/74:0][013_partition.pl] LOG: statement: ALTER SUBSCRIPTION
sub2 SET PUBLICATION pub_lower_level, pub_all
2022-03-18 01:33:15.354 CET [2551193][logical replication
worker][4/57:0][] LOG: logical replication apply worker for
subscription "sub2" has started
2022-03-18 01:33:15.605 CET [2551176][client
backend][:0][013_partition.pl] LOG: disconnection: session time:
0:00:00.415 user=bf database=postgres host=[local]
2022-03-18 01:33:15.607 CET [2551209][logical replication
worker][3/76:0][] LOG: logical replication table synchronization
worker for subscription "sub2", table "tab4_1" has started
2022-03-18 01:33:15.609 CET [2551211][logical replication
worker][5/11:0][] LOG: logical replication table synchronization
worker for subscription "sub2", table "tab3" has started
2022-03-18 01:33:15.617 CET [2551193][logical replication
worker][4/62:0][] LOG: logical replication apply worker for
subscription "sub2" will restart because of a parameter change

You will notice that the apply worker is never restarted after a
parameter change. The reason was that the particular subscription
reaches the limit of max_sync_workers_per_subscription after which we
don't allow to restart the apply worker. I think you might want to
increase the values of
max_sync_workers_per_subscription/max_logical_replication_workers to
make it work.

> I'd like to test & polish the main patch over the weekend, and get it
> committed early next week. Unless someone thinks it's definitely not
> ready for that ...
>

I think it is in good shape but apart from cleanup, there are issues
with dependency handling which I have analyzed and reported as one of
the comments in the email [2]. I was getting some weird behavior
during my testing due to that. Apart from that still the patch has DDL
handling code in tablecmds.c which probably is not required.
Similarly, Shi-San has reported an issue with replica full in her
email [3]. It is up to you what to do here but it would be good if you
can once share the patch after fixing these issues so that we can
re-test/review it.

[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=pogona&dt=2022-03-17%2023%3A10%3A04
[2] - https://www.postgresql.org/message-id/CAA4eK1KR%2ByUQquK0Bx9uO3eb5xB1e0rAD9xKf-ddm5nSf4WfNg%40mail.gmail.com
[3] - https://www.postgresql.org/message-id/TYAPR01MB6315D664D926EF66DD6E91FCFD109%40TYAPR01MB6315.jpnprd01.prod.outlook.com

--
With Regards,
Amit Kapila.