From: | Dave Cramer <davecramer(at)gmail(dot)com> |
---|---|
To: | Andres Freund <andres(at)anarazel(dot)de> |
Cc: | Petr Jelinek <petr(dot)jelinek(at)2ndquadrant(dot)com>, David Fetter <david(at)fetter(dot)org>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: Binary support for pgoutput plugin |
Date: | 2019-06-05 22:47:57 |
Message-ID: | CADK3HHKt7SeEM95O6F3Bf70CP6x_QdN4Y7PrZR16bw9tPmrP7Q@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Hi,
On Wed, 5 Jun 2019 at 12:01, Andres Freund <andres(at)anarazel(dot)de> wrote:
> Hi
>
> On June 5, 2019 8:51:10 AM PDT, Dave Cramer <davecramer(at)gmail(dot)com> wrote:
> >On Wed, 5 Jun 2019 at 07:21, Dave Cramer <davecramer(at)gmail(dot)com> wrote:
> >
> >> Hi,
> >>
> >>
> >> On Wed, 5 Jun 2019 at 07:18, Petr Jelinek
> ><petr(dot)jelinek(at)2ndquadrant(dot)com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> On 05/06/2019 00:08, Andres Freund wrote:
> >>> > Hi,
> >>> >
> >>> > On 2019-06-05 00:05:02 +0200, David Fetter wrote:
> >>> >> Would it make sense to work toward a binary format that's not
> >>> >> architecture-specific? I recall from COPY that our binary format
> >is
> >>> >> not standardized across, for example, big- and little-endian
> >machines.
> >>> >
> >>> > I think you recall wrongly. It's obviously possible that we have
> >bugs
> >>> > around this, but output/input routines are supposed to handle a
> >>> > endianess independent format. That usually means that you have to
> >do
> >>> > endianess conversions, but that doesn't make it non-standardized.
> >>> >
> >>>
> >>> Yeah, there are really 3 formats of data we have, text protocol,
> >binary
> >>> network protocol and internal on disk format. The internal on disk
> >>> format will not work across big/little-endian but network binary
> >>> protocol will.
> >>>
> >>> FWIW I don't think the code for binary format was included in
> >original
> >>> logical replication patch (I really tried to keep it as minimal as
> >>> possible), but the code and protocol is pretty much ready for adding
> >that.
> >>>
> >> Yes, I looked through the public history and could not find it.
> >Thanks for
> >> confirming.
> >>
> >>>
> >>> That said, pglogical has code which handles this (I guess Andres
> >means
> >>> that by predecessor of pgoutput) so if you look for example at the
> >>> write_tuple/read_tuple/decide_datum_transfer in
> >>>
> >>>
> >
> https://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical_proto_native.c
> >>> that can help you give some ideas on how to approach this.
> >>>
> >>
> >> Thanks for the tip!
> >>
> >
> >Looking at:
> >
> https://github.com/postgres/postgres/blob/8255c7a5eeba8f1a38b7a431c04909bde4f5e67d/src/backend/replication/pgoutput/pgoutput.c#L163
> >
> >this seems completely ignored. What was the intent?
>
> That's about the output of the plugin, not the datatypes. And independent
> of text/binary output, the protocol contains non-printable chars.
>
> Andres
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
So one of the things they would like added is to get not null information
in the schema record. This is so they can mark the field Optional in Java.
I presume this would also have some uses in other languages. As I
understand it this would require a protocol bump. If this were to be
accepted are there any outstanding asks that would useful to add if we were
going to bump the protocol?
Dave
From | Date | Subject | |
---|---|---|---|
Next Message | Melanie Plageman | 2019-06-05 22:49:47 | Re: Adding a test for speculative insert abort case |
Previous Message | Alvaro Herrera | 2019-06-05 21:54:08 | Re: Remove useless associativity/precedence from parsers |