From: | "Ed L(dot)" <pgsql(at)bluepolka(dot)net> |
---|---|
To: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
Cc: | pgsql-general(at)postgresql(dot)org, Steven Singer <ssinger(at)navtechinc(dot)com> |
Subject: | Re: Batch replication ordering (was Re: [GENERAL] 32/64-bit transaction IDs?) |
Date: | 2003-04-10 22:52:12 |
Message-ID: | 200304101652.12737.pgsql@bluepolka.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general |
On Thursday April 10 2003 4:44, Ed L. wrote:
> On Thursday April 10 2003 4:11, Tom Lane wrote:
> > "Ed L." <pgsql(at)bluepolka(dot)net> writes:
> > > On Saturday March 22 2003 12:00, Tom Lane wrote:
> > >> Note that all of a transaction's updates will become visible in the
> > >> pending-update table simultaneously when it commits, so (as long as
> > >> you grab batches in single SELECTs, or with a serializable
> > >> transaction) there's no problems with partial transactions being
> > >> applied by a batch.
> > >
> > > If you grab everything in the queue with a single SELECT, this works.
> > > Depending on the queue length, that's not always practical, and as
> > > noted above, committed batches could result in partial transactions
> > > on the slave. So the riddle is how to get a consistent but batchable
> > > replication order.
> >
> > You don't have to do anything special if you pull the contents of a
> > batch in a single serializable transaction. I see no reason to think
> > that using a serializable transaction is "hammering the master"; so
> > you are asking for a solution to a non-problem.
>
> I don't think so. Can you imagine a replication queue big enough to that
> someone might not want to process it entirely in one transaction? I sure
> can. Consider the following sequence:
>
> 1. Replication queueing is begun for a particular slave on the master.
> A dump is taken with which to initialize the slave.
>
> 2. A sufficient number of updates are queued such that the total amount
> of data exceeds the amount one wants to process before giving the master,
> or the slave, or the network, a break). This could easily happen in our
> case if there were delays in getting the slave setup for whatever reason.
By "sufficient number of updates", I really mean a large enough number of
transactions. Imagine a queue with a 100,000 transactions that would
require a number of hours or days to replicate straight through.
Ed
>
> 3. The slave is finally setup for replication. By this time, the queue
> is bigger than we want to process in one round.
>
> Ed
From | Date | Subject | |
---|---|---|---|
Next Message | Joseph Shraibman | 2003-04-10 23:11:46 | Re: The mail nttp gateway is still broken |
Previous Message | Ed L. | 2003-04-10 22:44:20 | Re: Batch replication ordering (was Re: [GENERAL] 32/64-bit transaction IDs?) |