Re: Catalog/Metadata consistency during changeset extraction from wal

Hi,

On Monday, June 25, 2012 08:13:54 PM Robert Haas wrote:
> On Mon, Jun 25, 2012 at 1:50 PM, Andres Freund <andres(at)2ndquadrant(dot)com>
wrote:
> > Its an argument why related infrastructure would be interesting to more
> > than that patch and thats not bad.
> > If the garbage collecting is done in a very simplistic manner it doesn't
> > sound too hard... The biggest problem is probably crash-recovery of that
> > knowledge and how to hook knowledge into it that logical rep needs that
> > data...
> I suppose the main reason we haven't done it already is that it
> increases the period of time during which we're using 2X the disk
> space.
I find that an acceptable price if its optional. Making it such doesn't seem
to be a problem for me.

> >> > I don't think its that easy. If you e.g. have multiple ALTER's in the
> >> > same transaction interspersed with inserted rows they will all have
> >> > different TupleDesc's.
> >>
> >> If new columns were added, then tuples created with those older
> >> tuple-descriptors can still be interpreted with the latest
> >> tuple-descriptor.
> >
> > But you need to figure that out. If you have just the before-after images
> > of the tupledescs you don't know what happened in there... That would
> > mean either doing special things on catalog changes or reassembling the
> > meaning from the changed pg_* rows. Neither seems enticing.
>
> I think there is absolutely nothing wrong with doing extra things in
> ALTER TABLE when logical replication is enabled. We've got code
> that's conditional on Hot Standby being enabled in many places in the
> system; why should logical replication be any different? If we set
> the bar for logical replication at "the system can't do anything
> differently when logical replication is enabled" then I cheerfully
> submit that we are doomed. You've already made WAL format changes to
> support logging the pre-image of the tuple, which is a hundred times
> more likely to cause a performance problem than any monkeying around
> we might want to do in ALTER TABLE.
>
> I am deeply skeptical that we need to look inside of transactions that
> do full-table rewrites. But even if we do, I don't see that what I'm
> proposing precludes it. For example, I think we could have ALTER
> TABLE emit WAL records specifically for logical replication that allow
> us to disentangle which tuple descriptor to use at which point in the
> transaction. I don't see that that would even be very difficult to
> set up.
Sorry, I was imprecise above: I have no problem doing some changes during
ALTER TABLE if logical rep is enabled. I am worried though that to make that
robust you would need loads of places that emit additional information:
* ALTER TABLE
* ALTER FUNCTIION
* ALTER OPERATOR
* ALTER/CREATE CAST
* TRUNCATE
* CLUSTER
* ...

I have the feeling that would we want to do that the full amount of required
information would be rather high and end up being essentially the catalog.
Just having an intermediate tupledesc doesn't help that much if you have e.g.
record_out doing type lookups of its own.

There also is the issue you have talked about before, that a user-type might
depend on values in other tables. Unless were ready to break at least
transactional behaviour for those for now...) I don't see how decoding outside
of the transaction is ever going to be valid? I wouldn't have a big problem
declaring that as broken for now...

Greetings,

Andres

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services