Re: Changeset Extraction Interfaces

On 2013-12-16 23:01:16 -0500, Robert Haas wrote:
> On Sat, Dec 14, 2013 at 12:37 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > On 2013-12-14 11:50:00 -0500, Robert Haas wrote:
> >> Well, it still seems to me that the right way to think about this is
> >> that the change stream begins at a certain point, and then once you
> >> cross a certain threshold (all transactions in progress at that time
> >> have ended) any subsequent snapshot is a possible point from which to
> >> roll forward.
> >
> > Unfortunately it's not possible to build exportable snapshots at any
> > time - it requires keeping far more state around since we need to care
> > about all transactions, not just transactions touching the
> > catalog. Currently you can only export the snapshot in the one point we
> > become consistent, after that we stop maintaining that state.
>
> I don't get it. Once all the old transactions are gone, I don't see
> why you need any state at all to build an exportable snapshot. Just
> take a snapshot.

The state we're currently decoding, somewhere in already fsynced WAL,
won't correspond to the state in the procarray. There might be
situations where it will, but we can't guarantee that we ever reach that
point without taking locks that will be problematic.

> The part that you're expressing willingness to do sounds entirely
> satisfactory to me. As I mentioned on the other thread, I'm perhaps
> even willing to punt that feature entirely provided that we have a
> clear design for how to add it later, but I think it'd be nicer to get
> it done now.

We'll see how the next version looks like. Not sure on that myself yet
;)

> And just for the record, I think the idea that I am holding this patch
> hostage is absurd. I have devoted a large amount of time and energy
> to moving this forward and plan to devote more. Because of that work,
> big chunks of what is needed here are already committed. If my secret
> plan is to make it as difficult as possible for you to get this
> committed, I'm playing a deep game.

I am not saying at all that you're planning to stop the patch from
getting in. You've delivered pretty clear proof that that's not the
case.
But that doesn't prevent us from arguing over details and disagreeing
whether they are dealbreakers or not, does it ;)

I think you know that I am hugely grateful for the work you've put into
the topic.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services