Re: Detecting schema changes during logical replication

On 8 May 2017 05:56, "Daniele Varrazzo" <daniele(dot)varrazzo(at)gmail(dot)com> wrote:

On Sun, May 7, 2017 at 8:04 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> Hi,
>
> On 2017-05-07 19:27:08 +0100, Daniele Varrazzo wrote:
>> I'm putting together a replication system based on logical
>> replication.
>
> Interesting. If you very briefly could recap what it's about... ;)

I need to replicate some tables from a central database into the
database that should run a secondary system. For a similar use case we
have used Londiste in the past, which has served us good, but its
usage has not been problem-free. Logical decoding seems much less
invasive on the source database than a trigger-based replication
solution, and has less moving part to care about and maintain.

For the moment I'm hacking into a fork of Euler project for wal
decoding into json (https://github.com/dvarrazzo/wal2json) mostly
adding configurability, so that we may be able to replicate only the
tables we need, skip certain fields etc. I'm also taking a look at
minimising the amount of information produced: sending over and over
the column names and types for every record seems a waste, hence my
question.

Sounds like you're reimplementing pglogical (
http://2ndquadrant.com/pglogical) on top of a json protocol.

Pglogical is open source and hackable to meet your needs. We're also happy
to accept patches with appropriate design discussion and code review to
make sure they will aid or not hinder other users and won't add undue
maintenance burden.

Rather than repeat the same work, maybe it's worth picking it up as a
starting point. It's the extension-based progenitor of the in-core logical
rep code, but portable from 9.4 to 9.6 (and soon 10) with a bunch of
features that didn't make it into Pg's version yet.

I have no reason to object to your doing it yourself, and you're welcome to
use pglogical as a reference for how to do things (see the license). It
just seems like a waste.