Re: Logical replication and multimaster

On Tue, 15 Dec 2015 21:48:52 -0600
Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com> wrote:

> On 12/13/15 7:37 AM, David Fetter wrote:
> > As I understand it, pushing these into a library has been proposed but
> > not rejected. That it hasn't happened yet is mostly about the lack of
> > tuits (the round ones) to rewrite the functionality as libraries and
> > refactor pg_dump/pg_restore to use only calls to same. As usual, it's
> > less about writing the code and more about the enormous amount of
> > testing any such a refactor would entail.
>
> My understanding as well. IIRC Jon Erdman brought this question up a
> couple years ago and the response was "It'd probably be accepted, it's
> just that no one has done the work."

To be clear, if I understand what you are referring to, my name pops up in this discussion because in Amsterdam around the time that FDWs were originally launched, I had a "brilliant" idea that a great candidate for an FDW would be one that could read from a custom dump (-Fc) file and expose it as though it contained regular tables, so you could restore one erroneously deleted row from a 2TB dump without loading the whole thing, or the whole table in question.

On the face of it this seemed relatively simple since a custom dump has a TOC and all the requisite goodies to make this doable, plus the code exists to interpret that (for restoring just one table out of a dump file) and all that was needed was the "glue" to hook it into FDW.

Initially the reaction (from Magnus if I'm not mistaken) was "that's stupid, who would want that", but later Dave Page was wholly on board with it.

At the next pgcon I spoke up on the same subject at the end of a talk about FDWs where Tom was in attendance, and all agreed my idea had merit...however, unexpectedly they (including Tom) agreed that trying to turn that part of our command line functionality into a library (the proper solution) was more effort than it was worth, and that if I wanted to try it I should just cut and paste the relevant code out of pg_dump and into my FDW, rather than trying to refactor and share said code in a .so. [I was *VERY* surprised by this!]

No one said it couldn't be done, but even the "wise men on the mount" conceded that it was such a huge undertaking that it was not worth the effort, and duplicating and subsequently maintaining said duplicated code was the better part of valor.

> > I believe that refactoring much of pg_dump's functionality for the
> > current version of the server into SQL-accessible functions and making
> > pg_dump use only those functions is achievable with available
> > resources.
> >
> > Such a refactor need not be all-or-nothing. For example, the
> > dependency resolution stuff is a first step that appears to be worth
> > doing by itself even if the effort then pauses, possibly for some
> > time.
>
> If someone wanted to spend time on this, I suspect it'd be worth looking
> at how bad some of the backward compatibility issues would be if done in
> the server. Maybe they wouldn't be that bad. I suspect the audience for
> this code would be much larger if it was in the server as opposed to a C
> library.
> --
> Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
> Experts in Analytics, Data Architecture and PostgreSQL
> Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Jon Erdman (aka StuckMojo)
PostgreSQL Zealot