Re: Logical Replication of sequences

On Thu, Jun 6, 2024 at 9:32 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Wed, Jun 5, 2024 at 12:43 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Tue, Jun 4, 2024 at 8:56 PM Yogesh Sharma
> > <yogesh(dot)sharma(at)catprosystems(dot)com> wrote:
> > >
> > > On 6/4/24 06:57, Amit Kapila wrote:
> > >
> > > > 2. Provide a command say Alter Subscription ... Replicate Sequences
> > > > (or something like that) which users can perform before shutdown of
> > > > the publisher node during upgrade. This will allow copying all the
> > > > sequences from the publisher node to the subscriber node directly.
> > > > Similar to previous approach, this could also be inconvenient for
> > > > users.
> > >
> > > This is similar to option 1 except that it is a SQL command now.
> > >
> >
> > Right, but I would still prefer a command as it provides clear steps
> > for the upgrade. Users need to perform (a) Replicate Sequences for a
> > particular subscription (b) Disable that subscription (c) Perform (a)
> > and (b) for all the subscriptions corresponding to the publisher we
> > want to shut down for upgrade.
> >
> > I agree there are some manual steps involved here but it is advisable
> > for users to ensure that they have received the required data on the
> > subscriber before the upgrade of the publisher node, otherwise, they
> > may not be able to continue replication after the upgrade. For
> > example, see the "Prepare for publisher upgrades" step in pg_upgrade
> > docs [1].
> >
> > >
> > > > 3. Replicate published sequences via walsender at the time of shutdown
> > > > or incrementally while decoding checkpoint record. The two ways to
> > > > achieve this are: (a) WAL log a special NOOP record just before
> > > > shutting down checkpointer. Then allow the WALsender to read the
> > > > sequence data and send it to the subscriber while decoding the new
> > > > NOOP record. (b) Similar to the previous idea but instead of WAL
> > > > logging a new record directly invokes a decoding callback after
> > > > walsender receives a request to shutdown which will allow pgoutput to
> > > > read and send required sequences. This approach has a drawback that we
> > > > are adding more work at the time of shutdown but note that we already
> > > > waits for all the WAL records to be decoded and sent before shutting
> > > > down the walsender during shutdown of the node.
> > >
> > > At the time of shutdown a) most logical upgrades don't necessarily call
> > > for shutdown
> > >
> >
> > Won't the major version upgrade expect that the node is down? Refer to
> > step "Stop both servers" in [1].
>
> I think the idea is that the publisher is the old version and the
> subscriber is the new version, and changes generated on the publisher
> are replicated to the subscriber via logical replication. And at some
> point, we change the application (or a router) settings so that no
> more transactions come to the publisher, do the last upgrade
> preparation work (e.g. copying the latest sequence values if
> requried), and then change the application so that new transactions
> come to the subscriber.
>

Okay, thanks for sharing the exact steps. If one has to follow that
path then sending incrementally (at checkpoint WAL or other times)
won't work because we want to ensure that the sequences are up-to-date
before the application starts using the new database. To do that in a
bullet-proof way, one has to copy/replicate sequences during the
requests to the new database are paused (Reference from the blog you
shared: For the first second after flipping the flag, our application
artificially paused any new database requests for one second.).
Currently, they are using some guesswork to replicate sequences that
require manual verification and more manual work for each sequence.
The new command (Alter Subscription ... Replicate Sequence) should
ease their procedure and can do things where they would require no or
very less verification.

> I remember the blog post about Knock doing a similar process to
> upgrade the clusters with minimal downtime[1].
>

Thanks for sharing the blog post.

--
With Regards,
Amit Kapila.