Re: Built-in plugin for logical decoding output

On 25/09/17 21:38, Andres Freund wrote:
> On 2017-09-25 21:31:11 +0300, Alvaro Hernandez wrote:
>>>>> - Distribution and testing are non-trivial: many OS/archs combinations.
>>>>>
>>> Yes, it is. Why would we want to increase that burden to this community?
>>
>>     That's a different story, and one I cannot argue against. If easying
>> postgresql integration with other tools is not something of a priority or
>> something the core group cannot add to all the stuff on their shoulders, all
>> my due respect. PostgreSQL users will do without, someway, somehow. But IMHO
>> this should be a really high priority, and saying that this would turn
>> PostgreSQL into an Oracle code base is going too far ;)
> Well, we can certainly use more help doing maintenance-y stuff, which
> will in turn allow to get more stuff into core in a good state medium+
> term... ;)
>
> Less jokingly: I'm doubtful that the "not a priority" take is really
> fair - there's a lot of priorities, and they compete for scant
> resources. Which means people have to argue convincingly if they want to
> add to that burden - just actually asking the question whether it's a
> good use of resources doesn't mean it's not. Just that it's not yet
> clear.
>
>

    OK, let me try to do that. I believe data integration is a
priority. World is no longer an isolated database where many apps talk
to it. Nowdays heterogeneous architectures are almost the norm. CDC is
often the best solution for many of the data integration tasks like data
warehousing, data consolidation, stream processing, etc. From this
perspective, it would be key to have a general tool or good starting
point for CDC or even higher level functionality tools. Think of
PostgreSQL's Golden Gate.

    Now, developing any software like this faces two significant
challenges, which turn into deterrents for developing such software:

- If you want to develop your own output plugin, then your market is
reduced as you have to exclude all the managed solutions (until, and
only if, you would convince them to include your plugin... highly
unlikely, very difficult). And probably another % of enterprise
environments which will hesitate to link your own plugin to the running
production database. Last but not least, you need to compile and test
(and testing this is far from easy) on multiple OSes/architectures.

- If you stick to in-core plugins, then you need to support at least
three different output formats if you want to support 9.4+:
test_decoding (and pray it works!), pgoutput, and the "new" in-core
plugin that was proposed at the beginning of this thread, if that would
see the light.

    Both are strong enough arguments to make building this kind
software far less interesting. Actually, I don't know of any software
like this, and this may already be a consequence of what I'm saying. And
I truly believe PostgreSQL should offer this, as part of its ecosystem
(not necessarily in core).

    On the other hand, in-core may help encourage building this
solutions. If there would be an in-core, uniform, flexible, output
plugin, and ideally backported to 9.4 (I know, I know....), included in
very PostgreSQL... it would really open the doors to many integration
applications. Output plugins are great for your own use, for controlled
environments, for specific set of users. But only in-core plugins help
for general-purpose solutions.

    Álvaro

--

Alvaro Hernandez

-----------
OnGres