Re: Changeset Extraction Interfaces

On Sat, Dec 14, 2013 at 12:37 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> On 2013-12-14 11:50:00 -0500, Robert Haas wrote:
>> On Fri, Dec 13, 2013 at 9:14 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>> > But can you imagine those users needing an exported snapshot? I can
>> > think of several short-lived usages, but all of those are unlikely to
>> > need a consistent view of the overall database. And those are less
>> > likely to be full blown replication solutions.
>> > I.e. it's not the DBA making that decision but the developer making the
>> > decision based on whether he requires the snapshot or not.
>>
>> Well, it still seems to me that the right way to think about this is
>> that the change stream begins at a certain point, and then once you
>> cross a certain threshold (all transactions in progress at that time
>> have ended) any subsequent snapshot is a possible point from which to
>> roll forward.
>
> Unfortunately it's not possible to build exportable snapshots at any
> time - it requires keeping far more state around since we need to care
> about all transactions, not just transactions touching the
> catalog. Currently you can only export the snapshot in the one point we
> become consistent, after that we stop maintaining that state.

I don't get it. Once all the old transactions are gone, I don't see
why you need any state at all to build an exportable snapshot. Just
take a snapshot.

>> 1. Slots. We know we need physical slots as well as logical slots,
>> but the patch as currently constituted only offers logical slots.
>
> Well, then tell me the way you want to go forward on that end. I can
> make the slot interface more generic if we know exactly what we need,
> but I doesn't seem fair to take this patch hostage until I develop a
> separate not so small feature. Why is that my task?
> Because I think it's important, and because by now I know the related
> code pretty well by now, I am willing to provide the parts of the that
> prevent required WAL from being deleted, peg xmin and report the current
> state to SQL, but somebody else is going to have to the rest.

The part that you're expressing willingness to do sounds entirely
satisfactory to me. As I mentioned on the other thread, I'm perhaps
even willing to punt that feature entirely provided that we have a
clear design for how to add it later, but I think it'd be nicer to get
it done now.

And just for the record, I think the idea that I am holding this patch
hostage is absurd. I have devoted a large amount of time and energy
to moving this forward and plan to devote more. Because of that work,
big chunks of what is needed here are already committed. If my secret
plan is to make it as difficult as possible for you to get this
committed, I'm playing a deep game.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company