Re: Changeset Extraction Interfaces

On 2013-12-13 08:30:41 -0500, Robert Haas wrote:
> On Thu, Dec 12, 2013 at 1:52 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > Puh. I honestly have zero confidence in DBAs making an informed decision
> > about something like this. Honestly, for a replication solution, how
> > often do you think this will be an issue?
>
> If you imagine a scenario where somebody establishes a replication
> slot and then keeps it forever, not often. But if you're trying to do
> something more ad hoc, where replication slots might be used just for
> short periods of time and then abandoned, I think it could come up
> pretty frequently.

But can you imagine those users needing an exported snapshot? I can
think of several short-lived usages, but all of those are unlikely to
need a consistent view of the overall database. And those are less
likely to be full blown replication solutions.
I.e. it's not the DBA making that decision but the developer making the
decision based on whether he requires the snapshot or not.

> Generally, I think you're being too dismissive of the stuff I'm
> complaining about here. If we just can't get this, well then I
> suppose we can't.

I think I am just scared of needing to add more features before getting
the basics done and in consequence overrunning 9.4...

> But I think the amount of time that it takes
> Hot Standby to open for connections is an issue, precisely because
> it's got to wait until certain criteria are met before it can
> establish a snapshot, and sometimes that takes an unpleasantly long
> time.

The HS situation is quite different though. There the problem is hit on
every restart, not just the initial start; Suboverflowed xl_running
xacts can delay HS startup for a long time; and we don't start up at the
exact point we know a xl_running_xacts just has been logged.

FWIW I think we need to fix HS to store it's visibility state at
restartpoints, so we get rid of that problem. Not sure who will have
time to do that tho.

> That having been said, I don't find it at all
> hard to imagine someone wanting to jump into the replication stream at
> an arbitrary point in time and see changes from every transaction that
> *commits* after that point, even if it began earlier, or even to see
> changes from transactions that have not yet committed as they happen.

I agree they are desirable, but all those will require additional state
to be kept about running transactions. That's not saying it
shouldn't/cannot be done, to the contrary, just that it requires some
engineering effort.

> I realize that's asking for a pony, and I'm not saying you have to go
> off and do that right now in order for this to move forward, or indeed
> that it will ever happen at all. What I am saying is that I find it
> entirely likely that people are going to push the limits of this
> thing, that this is one of the limits I expect them to push, and that
> the more we can do to put policy in the hands of the user without
> pre-judging the sanity of what they're trying to do, the happier we
> (and our users) will be.

Completely agreed. I really think this the most basic building block,
missing many important features. And we'll be busy for some time adding
those.

> > One point for my suggested behaviour is that it closes a bigger
> > racecondition. Currently as soon as start_logical_replication() has
> > finished building the tuplestore it marks the endposition as
> > received. But we very well can fail before the user has received all
> > those changes.
>
> Right. I think your idea is good, but maybe there should also be a
> version of the function that never confirms receipt even if the
> transaction commits. That would be useful for ad-hoc poking at the
> queue.

Ok, that sounds easy enough, maybe
pg_decoding_slot_get_[binary_]changes()
pg_decoding_slot_peek_[binary_]changes()
?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services