Re: [PATCH 10/16] Introduce the concept that wal has a 'origin' node

On Tuesday, June 19, 2012 07:24:13 PM Robert Haas wrote:
> On Tue, Jun 19, 2012 at 12:11 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> >> On Tuesday, June 19, 2012 04:30:59 PM Tom Lane wrote:
> >>>> ... (If you are thinking
> >>>> of something sufficiently high-level that merging could possibly work,
> >>>> then it's not WAL, and we shouldn't be trying to make the WAL
> >>>> representation cater for it.)
> >>
> >> Do you really see this as such a big problem?
> >
> > It looks suspiciously like "I have a hammer, therefore every problem
> > must be a nail". I don't like the design concept of cramming logical
> > replication records into WAL in the first place.
>
> Me, neither. I think it's necessary to try to find a way of
> generating logical replication records from WAL. But once generated,
> I think those records should form their own stream, independent of
> WAL. If you take the contrary position that they should be included
> in WAL, then when you filter the WAL stream down to just the records
> of interest to logical replication, you end up with a WAL stream with
> holes in it, which is one of the things that Andres listed as an
> unresolved design problem in his original email.
Yes.

> Moreover, this isn't necessary at all for single-master replication,
> or even multi-source replication where each table has a single master.
> It's only necessary for full multi-master replication, which we have
> no consensus to include in core, and even if we did have a consensus
> to include it in core, it certainly shouldn't be the first feature we
> design.
Well, you can't blame a patch/prototype trying to implement what it claims to
implement ;)

More seriously: Even if we don't put MM in core I think putting the basis for
it in core so that somebody can build such a solution reusing the existing
infrastructure is a sensible idea. Imo the only thing that requires explicit
support which is hard to add outside of core is prevention of loops (aka this
patch). Everything else should be doable reusing the hopefully modular pieces.

> > However, if we're dead set on doing it that way, let us put information
> > that is only relevant to logical replication records into only the
> > logical replication records.
> Right. If we decide we need this, and if we did decide to conflate
> the WAL stream, both of which I disagree with as noted above, then we
> still don't need it on every record. It would probably be sufficient
> for local transactions to do nothing at all (and we can implicitly
> assume that they have master node ID = local node ID) and transactions
> which are replaying remote changes to emit one record per XID per
> checkpoint cycle containing the remote node ID.
Youve gone from a pretty trivial 150 line patch without any runtime/space
overhead to something *considerably* more complex in that case though.

You suddently need to have relatively complex logic to remember which
information you got for a certain xid (and forget that information afterwards)
and whether you already logged that xid and you need to have to decide about
logging that information at multiple places.

Btw, what do you mean with "conflating" the stream? I don't really see that
being proposed.

Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services