Re: logical changeset generation v6

On 2013-09-24 11:04:06 -0400, Robert Haas wrote:
> - Requiring a client is a short-sighted design. There's no reason we
> shouldn't *support* having a client, but IMHO it shouldn't be the only
> way to use the feature.

There really aren't many limitations preventing you from doing anything
else.

> - Suppose that you use pg_receivellog (or whatever we decide to call
> it) to suck down logical replication messages. What exactly are you
> going to do with that data once you've got it? In the case of
> pg_receivexlog it's quite obvious what you will do with the received
> files: you'll store them in archive of some kind and maybe eventually
> use them for archive recovery, streaming replication, or PITR. But
> the answer here is a lot less obvious, at least to me.

Well, it's not like it's going to be the only client. But it's a useful
one. I don't see this as an argument against pg_receivelogical? Most
sites don't use pg_receivexlog either.
Not having a consumer of the walsender interface included sounds like a
bad idea to me, even if it were only useful for testing. Now, you could
argue it should be in /contrib - and I wouldn't argue against that
except it shares code with the rest of src/bin/pg_basebackup.

> > If somebody needs something like this, e.g. because they want to
> > replicate into hundreds of shards depending on some key or such, the
> > question I don't know is how to actually initiate the
> > streaming. Somebody would need to start the logical decoding.

> Sounds like a job for a background worker. It would be pretty swell
> if you could write a background worker that connects to a logical
> replication slot and then does whatever.

That's already possible. In that case you don't have to connect to a
walsender, although doing so would give you some parallelism, one
decoding the data, the other processing it ;).

There's one usecase I do not forsee decoupling from the walsender
interface this release though - synchronous logical replication. There
currently is no code changes required to make sync rep work for this,
and decoupling sync rep from walsender is too much to bite off in one
go.

> >> for materialized views, we might like to push the changes into delta
> >> relations within the source database.
> >
> > Yes, that's not a bad usecase and I think the only thing missing to use
> > output plugins that way is a convenient function to tell up to where
> > data has been received (aka synced to disk, aka applied).
>
> Yes. It feels to me (and I only work here) like the job of the output
> plugin ought to be to put the data somewhere, and the replication code
> shouldn't make too many assumptions about where it's actually going.

The output plugin just has two functions it calls to send out data,
'prepare_write' and 'write'. The callsite has to provide those
callbacks. Two are included. walsender and an SQL SRF.

Check the 'test_logical_decoding commit, it includes the SQL consumer.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services