Re: logical decoding documentation?

Hi,

On 2014-03-11 15:57:39 -0400, Peter Eisentraut wrote:
> Where, if anywhere, is the current documentation for writing or using a
> logical decoding output plugin consumer thingy?

There's a pending patch for it. The corresponding commit is
http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=commit;h=5eeedd55b2d7e53b5fdcdab6a8e74bb666d75bcc

I welcome feedback about it. I've spent a fair bit of time immersed in
this stuff, and I am not really sure anymore what's understandable and
whatnot ;)

It's referencing pg_recvlogical which isn't committed yet (that's the
commit just before), that's why the docs weren't committed with the
feature itself.

> src/backend/replication/logical/logical.c, which textually contains most
> of the functions that appear to interact with the test_decoding module,
> contains this in the header comment:
>
> """
> The idea is that a consumer provides three callbacks, one to read WAL,
> one to prepare a data write, and a final one for actually writing since
> their implementation depends on the type of consumer. Check
> logicalfunc.c for an example implementations of a fairly simple consumer
> and a implementation of a WAL reading callback that's suitable for
> simpler consumers.
> """
>
> There is no file logicalfunc.c.

Hrmpf: There's a missing 's', it's logicalfuncs.c.

> And test_decoding actually uses five callbacks, not three.

The callbacks logicalfuncs.c header comment is talking about adding a
new method to output data. Currently you can stream out changes via
walsender and via the SQL SRFs. But it might be interesting to
e.g. consume the changes in a bgworker, without going through either SQL
or walsender. To do that you need the three callbacks referenced above.

> Is a consumer the same as a decoder?

A consumer is just the recipient of the changestream. I.e the walsender
that streams out the changestream, or the SRF that spills the data into
a tuplestore.

I don't think the term "decoder" is used anywhere, but if it is, it'd
be the output plugin.

> test_decoding.c contains this:
>
> /* These must be available to pg_dlsym() */
> static void pg_decode_startup(LogicalDecodingContext *ctx,
> OutputPluginOptions *opt, bool is_init);
> ...
>
> which is surely wrong.

Hm, these days the comment should be above _PG_init() and
_PG_output_plugin_init(). That changed around several times...

Could you perhaps commit the attached patch fixing the issues you
mentioned?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services