Logical replication troubles

Hi,

I have run into a (to me) weird issue with logical replication. We are
running Zalandos postgres-operator in our Kubernetes clusters and have
recently had a use-case where we wanted to start doing logical replication
of select tables to a data warehouse, also running postgres. It worked as
expected at first, but then after a pod-restart in Kubernetes, the
replication slots that were created for the subscription were gone. A bit
of reading later, and I learn we need to tell Patroni which slots should be
permanently available, so we specify a slot and try to set this up, but
then run into an error which says the publication does not exist, even
though we can verify that it does. At first I suspected Patroni handling
the replication slots to be the cause of the problem, but about a week's
worth of learning and experimenting later, I can now reliably replicate the
problem in pure postgres. Patroni is kind of the catalyst, since my
findings are that if the replication slot is created before data is
inserted into the source database, and a publication is created, then it
breaks. If the replication slot is created after data is inserted and the
publication is created, then it works. We just can't tell Patroni to not
create it until some arbitrary point in time. I am guessing this is either
a bug or a case of us not knowing what we are doing...

I have created a Github gist demonstrating the problem:
https://gist.github.com/drzero42/02b4082ce002c1d90ddd64f5fe03aee0

Anybody able to help? :)

--
Anders Bøgh Bruun

Infrastructure Architect

CellPoint digital
cellpointdigital.com
*WE MAKE TRAVEL EASIER™*

M: +45 31 14 87 41
E: anders(at)cellpointdigital(dot)com

Chicago | *Copenhagen* | Dubai | London | Miami | Pune | Singapore