Re: Catalog/Metadata consistency during changeset extraction from wal

On Thursday, June 21, 2012 05:25:41 PM Simon Riggs wrote:
> On 21 June 2012 16:13, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > On Thursday, June 21, 2012 05:05:04 PM Simon Riggs wrote:
> >> On 21 June 2012 15:53, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> >> >> ISTM we should maintain a lookup table on target system that has the
> >> >> minimal required information on it.
> >> >
> >> > You need just about the whole catalog because the *_out procs might
> >> > need to lookup types, operators and such again.
> >> > Unless you want to rewrite those functions you need to provide a
> >> > normal execution environment.
> >>
> >> OK, so its more tables than I first thought, but its not all rows and
> >> columns of all catalog tables.
> >
> > Sure, there are a few you probably can leave out (pg_database, pg_auth*,
> > pg_*acl, pg_(sh)?depend, pg_database, pg_tablespace, ...) but its not
> > many.
>
> That's a start. Leaving out the shared catalogs makes me smile already.
>
> >> > I don't see how your idea works because of that? Am I missing
> >> > something?
> >>
> >> Why does the number/size of the tables required make that not work?
> >
> > The number of tables itself isn't a fundamental problem although it would
> > make stuff harder.
> > The problem is that the out functions expect a normal operating
> > environment and might e.g. do catalog lookups themselves. I don't see
> > how we can do anything here without providing a (nearly) full catalog.
>
> I accept that there could be pathological functions in there. We're
> not trying to make it work with any conceivable datatype/operator, so
> forcing logical replication to follow sensible rules makes sense. Are
> there any out functions that anybody uses that do that?
Loads. enum_out, record_out, array_out are examples I can think of without
even looking. I am pretty sure there are more. But imo this list already shows
its prohibitive.

> It's too much change to actually version the main catalog. Keeping a
> separate copy of a versioned catalog for use by replication sounds
> much more likely to fly.
I don't yet see how that should work given oids and everything are quite
possibly hardcoded in those functions. You could start switching out the
catalogs on a lower level but I think at that point its getting too ugly.

> In any case, I think we'll have to go back through the list and do
> more work on evaluation. When the options look like that, its typical
> to have ruled out the final winner early on, but that doesn't mean it
> isn't in there somewhere.
I hope we have but I am not convinced that there is an elegant solution...

Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services