Re: BUG #13725: Logical Decoding - wrong results with large transactions and unfortunate timing

On Mon, Oct 26, 2015 at 12:38 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:

> On 2015-10-26 12:34:48 +0100, Shulgin, Oleksandr wrote:
> > On Mon, Oct 26, 2015 at 12:30 PM, Andres Freund <andres(at)anarazel(dot)de>
> wrote:
> >
> > > On 2015-10-26 13:21:44 +0200, ofir(dot)manor(at)gmail(dot)com wrote:
> > > > Yes, this is a small script to reproduce, the real code is Java, we
> saw
> > > > sporadic wrong results.
> > > > However, I'm interested in CDC (get change notifications per row to
> my
> > > > app), not PG-to-PG replication.
> > >
> > > The streaming interface exists for row-by-row notification as well, and
> > > is a *LOT* more efficient.
> > >
> >
> > Yeah, but I don't think there's a workable Java implementation available
> > yet?
>
> No idea, but it's not that hard to write one.
>
> > > If there's a bug here, we obviously need to fix it nonetheless.
> > I would assume re-calculating end_of_wal in the while loop condition
> would
> > fix this?
>
> Why? That'd just lead to outputting more rows in one invocation, and
> that's it? I think I'm not following what you see as the problem?
>

I think there are just some false expectations involved about how this
interface should work. The OP likely expects that after the partial
results were returned by the first call to pg_logical_slot_get_changes(),
the next call will continue from the point where the first call left.

This doesn't happen because in the first call we never cross transaction
boundary? Hm, but why do we see the partial changes anyway? I would
assume if we started decoding this at all, the transaction was already
committed and end_of_wal will be past its end...

I'm lost.

--
Alex