Re: Built-in plugin for logical decoding output

On 24/09/17 02:41, Euler Taveira wrote:
> 2017-09-23 14:01 GMT-03:00 Alvaro Hernandez <aht(at)ongres(dot)com>:
>> However, AFAIK, AWS's DMS uses it for production purposes (see
>> http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html)
>>
> It seems a bad idea. AFAICS test_decoding was not designed to be a
> ready-for-production plugin. It is just a proof of concept for logical
> decoding.

    Yes, this is what I heard and read.

    However, if DMS uses it for what I'd call production use, I assume
it is actually production quality. I bet they do enough testing, and
don't ship software to potentially millions of customers if it doesn't
work well. So... first, I'd consider this a a sign of robustness.
Second..... my hats off for the plugin code ;)

>> I would be happy to see another logical decoding plugin into core
>> starting on 11. However, this also poses a bit of a challenge for middleware
>> implementors: you need to support one for 9.4-9.5 (test_decoding), another
>> for 10 (pgoutput) and maybe another for 11 onwards. The idea of asking users
>> to install a binary plugin is very unsexy, so these are the options
>> available.
>>
> wal2json works for 9.4+ (besides the WAL messages I committed a month
> ago). Since this boat was already shipped we can arrange some packages
> for 9.4-10 (an external project) and ask vendors to support the
> backward-compatible plugin. The middleware implementor will have to
> support this new plugin format. Being JSON a widespread format, it is
> easier to refactor the code to parse JSON.

    I agree its far better to parse JSON than the test_decoding output.
But asking any potential user to install a dynamic library, from a third
party website, which will need to be compiled for many potential
OSes/Archs, or even impossible if running on a managed environment... is
not a great experience. Unless PostgreSQL would backport a plugin and
ship it in newer releases, if test_decoding is good enough, I'd rather
stick to it.

>
>> However, having said that, and while json is a great output format for
>> interoperability, if there's a discussion on which plugin to include next,
>> I'd also favor one that has some more compact representation format (or that
>> supports several formats, not only json).
>>
> We could certainly extend pgoutput to support more than one format
> (like pglogical did AFAIR), however, we wouldn't reuse code (different
> formats) and will have a fat plugin (I don't foresee a plugin using
> different formats in the same connection. It is difficult to
> coordinate a change like that having only one-way communication).
>

    I think pgoutput is what it is. Maybe instead than growing it, my
+1 would be to add a new plugin that rather than being json only, would
also support other formats, like an efficient binary serialization.

    Álvaro

--

Alvaro Hernandez

-----------
OnGres