Re: [HACKERS] Built-in plugin for logical decoding output

Reviving this thread.

On Tue, 26 Sep 2017 at 13:57, Henry <henrymanmail(at)gmail(dot)com> wrote:

> It seems test_decoding.c could be easily changed to support JSON by using
> the built in PostgreSQL functions (json.c composite_to_json) to convert a
> Datum into SQL. It's use of OidOutputFunctionCall could be modified to emit
> arrays and composite types as JSON. This might be enough to enable
> downstream frameworks to parse (without having to code to the terse and
> positional composite structure format).
>
> It could be a minimal change to have in core using the built in JSON
> support with no additional libraries. I have not made changes to this code
> but it seems like it should work.
>
> Thank you,
> Henry
>
> On Tue, Sep 26, 2017 at 9:37 AM Alvaro Hernandez <aht(at)ongres(dot)com> wrote:
>
>>
>>
>> On 26/09/17 17:50, Craig Ringer wrote:
>>
>> On 26 September 2017 at 22:14, Magnus Hagander <magnus(at)hagander(dot)net>
>> wrote:
>>
>>>
>>>
>>> On Tue, Sep 26, 2017 at 2:16 PM, Alvaro Hernandez <aht(at)ongres(dot)com>
>>> wrote:
>>>
>>>>
>>>>
>>>>
>>>> But what about earlier versions? Any chance it could be backported
>>>> down to 9.4? If that would be acceptable, I could probably help/do that...
>>>
>>>
>>> The likelihood is zero if you mean backported into core of earlier
>>> versions.
>>>
>>
>> Right. We don't add features to back branches.
>>
>>
>> Yeah, I know the policy. But asking is free ;) and in my opinion this
>> would be a very good reason to have an exception, if there would be a clear
>> desire to have a single, unified, production quality output plugin across
>> all PostgreSQL versions. At least I tried ;)
>>
>>
>>
>>
>>
>>>
>>> If you mean backported as a standalone extension that could be installed
>>> on a previous version, probably. I'm not sure if it relies on any internals
>>> not present before that would make it harder, but it would probably at
>>> least be possible.
>>>
>>>
>> All the pub/sub stuff is new and hooked into syscache etc. So you'd be
>> doing a bunch of emulation/shims using user catalogs. Not impossible, but
>> probably irritating and verbose. And you'd have none of the DDL required to
>> manage it, so you'd need SQL-function equivalents.
>>
>> I suspect you'd be better off tweaking pglogical to speak the same
>> protocol as pg10, since the pgoutput protocol is an evolution of
>> pglogical's protocol. Then using pglogical on older versions.
>>
>>
>>
>> Given all this, if I would be doing an app based on logical decoding,
>> I think I will stick to test_decoding for <10....
>>
>>
>> Thanks,
>>
>>
>> Álvaro
>>
>>
>> --
>>
>> Alvaro Hernandez
>>
>>
>> -----------
>> OnGres
>>
>>

I believe there is a valid reason for providing a reasonably feature
complete plugin in core. Specifically in instances such as cloud providers
where the user does not control what is installed on the server it would be
useful to have a decent output plugin.

Having had a cursory look at pgoutput I see no reason why pgoutput could
not be used as general purpose output plugin.

One thing that would be nice is to remove the requirement for a publication
as creating a publication on all tables requires a superuser.
I'm also curious why pgoutput does not send attributes in binary ? This
seems like a rather small change that should provide some significant
performance benefits.

Dave Cramer

davec(at)postgresintl(dot)com
www.postgresintl.com