Re: Batch replication ordering (was Re: [GENERAL] 32/64-bit transaction IDs?)

On Thursday April 10 2003 4:44, Ed L. wrote:
> On Thursday April 10 2003 4:11, Tom Lane wrote:
> > "Ed L." <pgsql(at)bluepolka(dot)net> writes:
> > > On Saturday March 22 2003 12:00, Tom Lane wrote:
> > >> Note that all of a transaction's updates will become visible in the
> > >> pending-update table simultaneously when it commits, so (as long as
> > >> you grab batches in single SELECTs, or with a serializable
> > >> transaction) there's no problems with partial transactions being
> > >> applied by a batch.
> > >
> > > If you grab everything in the queue with a single SELECT, this works.
> > > Depending on the queue length, that's not always practical, and as
> > > noted above, committed batches could result in partial transactions
> > > on the slave. So the riddle is how to get a consistent but batchable
> > > replication order.
> >
> > You don't have to do anything special if you pull the contents of a
> > batch in a single serializable transaction. I see no reason to think
> > that using a serializable transaction is "hammering the master"; so
> > you are asking for a solution to a non-problem.
>
> I don't think so. Can you imagine a replication queue big enough to that
> someone might not want to process it entirely in one transaction? I sure
> can. Consider the following sequence:
>
> 1. Replication queueing is begun for a particular slave on the master.
> A dump is taken with which to initialize the slave.
>
> 2. A sufficient number of updates are queued such that the total amount
> of data exceeds the amount one wants to process before giving the master,
> or the slave, or the network, a break). This could easily happen in our
> case if there were delays in getting the slave setup for whatever reason.

By "sufficient number of updates", I really mean a large enough number of
transactions. Imagine a queue with a 100,000 transactions that would
require a number of hours or days to replicate straight through.

Ed

>
> 3. The slave is finally setup for replication. By this time, the queue
> is bigger than we want to process in one round.
>
> Ed