Re: Streaming replication for psycopg2

On 30 June 2015 at 22:42, Shulgin, Oleksandr
<oleksandr(dot)shulgin(at)zalando(dot)de> wrote:
> On Thu, Jun 4, 2015 at 5:49 PM, Shulgin, Oleksandr
> <oleksandr(dot)shulgin(at)zalando(dot)de> wrote:
>>
>> On Tue, Jun 2, 2015 at 2:23 PM, Shulgin, Oleksandr
>> <oleksandr(dot)shulgin(at)zalando(dot)de> wrote:
>> >
>> > I've submitted a patch to psycopg2 to support streaming replication
>> > protocol (COPY_BOTH): https://github.com/psycopg/psycopg2/pull/322
>
> Hello again,
>
> I have updated the pull request above to address the feedback I've gathered
> from using this construct internally and from other sources. Now included,
> the lower level asynchronous interface that gives the user more control, for
> the price of doing select() and keeping track of keepalive timeouts
> manually.
>
> It would be nice if someone could review this. The final look can be found
> by using this link (docs included):
> https://github.com/psycopg/psycopg2/pull/322/files

Per the pull request's history, I've given this a fairly detailed
review. With a few minor changes I think it's really good and will be
a valuable addition to psycopg2.

Before merging I think the connection should preferably be split into
a proper subclass, though it might not be worth the verbosity involved
when working with the CPython API.

I suspect that the row-by-row COPY support should be pushed down to be
shared with copy_expert, too. There's so much logic that'd be shared:
the push (callback) or pull oriented read modes, the support for
reading raw bytes or decoded unicode text for rows, the result object,
etc.

Otherwise I think this is great, and it was a real pleasure to read a
patch where the first half is good documentation and examples.

Since pyscopg2 is libpq based, so it can't be treated as an
independent re-implementation of the protocol for testing purposes.
For that it'd probably be necessary to add replication protocol
support to PgJDBC. But it's still really handy for prototyping logical
decoding clients, testing output plugins, adapting logical decoding
output to feed into other systems, etc, and I can see this as
something that could be really handy for optional extra tests for
PostgreSQL its self - validating the replication protocol, etc.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services