Re: viewing connectioninfo used by subscriber on the publication server when inactive

On Thu, May 14, 2020 at 10:22 AM Wim Bertels <wim(dot)bertels(at)ucll(dot)be> wrote:

> Keith Fiske schreef op do 14-05-2020 om 10:08 [-0400]:
> > It doesn't matter how small the dataset change is. The same WAL
> > stream is used for both logical and physical replication so it has to
> > keep all WAL files until all subscribers for that publication have
> > confirmed they have received them. If even a single subscriber goes
> > offline, all WAL will be kept until that subscriber reconnects.
>
> That is interesting, i assume this the WAL for the whole cluster, as
> logical decoding is then used on this WAL for the logical replication,
> do you have an estimate of order of magnitude for the all WAL files?
>
> So far this seems ok over here (with one subscriber inactive for 2
> days):
> # du -ch pg_logical/ pg_wal/ pg_replslot/
> 912K pg_logical/snapshots
> 4,0K pg_logical/mappings
> 924K pg_logical/
> 4,0K pg_wal/archive_status
> 81M pg_wal/
> 8,0K pg_replslot/db2_sub
> 8,0K pg_replslot/db2_sub1
> 8,0K pg_replslot/db2_sub2
> 28K pg_replslot/
> 81M totaal
> this after two days of replication setup.
>
> assuming that students will be offline for at most 2 or 3 days,
> this seems ok?
>
> --
> mvg,
> Wim
> --
> Tell the truth or trump--but get the trick.
> -- Mark Twain, "Pudd'nhead Wilson's Calendar"
>
>

WAL is always cluster/instance-wide. Locally generated WAL files are, by
default, always 16MB in size no matter how little has changed. So if you
have archive_timeout set and only a single row has changed, that WAL file
will still be 16MB (although it will compress down very small if you're
archiving elsewhere).

How much of an impact this will be is entirely dependent on the write rate
of your cluster. If you have very few writes you may be fine. But I would
definitely suggest getting some monitoring in place if you expect to have
offline subscribers for any long period of time.

But, again, I would still try and rethink this strategy. Offline
subscribers can be a very big problem if they never come back. Not only
because you'd eventually fill up your disk, but also because that no longer
allows PG to recycle its WAL files, or can cause excessive cleanup
operations when that subscriber finally comes back, which can have big IO
impacts.
--
Keith Fiske
Senior Database Engineer
Crunchy Data - http://crunchydata.com