Re: logical changeset generation v4

Hi,

I pushed a new rebased version (the xlogreader commit made it annoying
to merge).

The main improvements are
* way much coherent code internally for intializing logical rep
* explicit control over slots
* options for logical replication

On 2013-01-19 23:42:02 -0500, Steve Singer wrote:
> On 13-01-14 08:38 PM, Andres Freund wrote:
> >2) Currently the logical replication infrastructure assigns a 'slot-id'
> >when a new replica is setup. That slot id isn't really nice
> >(e.g. "id-321578-3"). It also requires that [18] keeps state in a global
> >variable to make writing regression tests easy.
> >
> >I think it would be better to make the user specify those replication
> >slot ids, but I am not really sure about it.
>
> Shortly after trying out the latest version I hit the following scenario
> 1. I started pg_receivellog but mistyped the name of my plugin
> 2. It looped and used up all of my logical replication slots
>
> I killed pg_receivellog and restarted it with the correct plugin name but it
> won't do anything because I have no free slots. I can't free the slots with
> -F because I have no clue what the names of the slots are. I can figure
> the names out by looking in pg_llog but if my replication program can't do
> that so it won't be able to clean up from a failed attempt.
>
> I agree with you that we should make the user program specify a slot, we
> eventually might want to provide a view that shows the currently allocated
> slots. For a logical based slony I would just generate the slot name based
> on the remote node id. If walsender generates the slot name then I would
> need to store a mapping between slot names and slons so when a slon
> restarted it would know which slot to resume using. I'd have to use a
> table in the slony schema on the remote database for this. There would
> always be a risk of losing track of a slot id if the slon crashed after
> getting the slot number but before committing the mapping on the remote
> database.

This is changed now, slotnames need to be provided and there also is a
pg_stat_logical_replication view (thanks Abhijit!).

> >3) Currently no options can be passed to an output plugin. I am thinking
> >about making "INIT_LOGICAL_REPLICATION 'plugin'" accept the now widely
> >used ('option' ['value'], ...) syntax and pass that to the output
> >plugin's initialization function.
>
> I think we discussed this last CF, I like this idea.

Added to the extension and walsender interface. Its used in the few
tests we have to specify that xids should not be included in the tests
for reproducability, so its even tested ;)

I haven't added code for setting up options via pg_receivellog yet.

> >5) Currently its only allowed to access catalog tables, its fairly
> >trivial to extend this to additional tables if you can accept some
> >(noticeable but not too big) overhead for modifications on those tables.
> >
> >I was thinking of making that an option for tables, that would be useful
> >for replication solutions configuration tables.
>
> I think this will make the life of anyone developing a new replication
> system easier. Slony has a lot of infrastructure for allowing slonik
> scripts to wait for configuration changes to popogate everywhere before
> making other configuration changes because you can get race conditions. If
> I were designing a new replication system and I had this feature then I
> would try to use it to come up with a simpler model of propagating
> configuration changes.

Working on it now.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services