Re: The plan for FDW-based sharding

On 28 February 2016 at 06:38, Kevin Grittner <kgrittn(at)gmail(dot)com> wrote:

>
> > For logical replay, applying in batches is actually a good thing since it
> > allows parallelism. We can remove them all from the target's procarray
> all
> > at once to avoid intermediate states becoming visible. So that would be
> the
> > preferred mechanism.
>
> That could be part of a solution. What I sketched out with the
> "apparent order of execution" ordering of the transactions
> (basically, commit order except when one SERIALIZABLE transaction
> needs to be dragged in front of another due to a read-write
> dependency) is possibly the simplest approach, but batching may
> well give better performance.
>

I'd be really interested in some ideas on how that information might be
usefully accessed. If we could write info on when to apply commits to the
xlog in serializable mode that'd be very handy, especially when looking to
the future with logical decoding of in-progress transactions, parallel
apply, etc.

For parallel apply I anticipated that we'd probably have workers applying
xacts in parallel and committing them in upstream commit order. They'd
sometimes deadlock with each other; when this happened all workers whose
xacts committed after the first aborted xact would have to abort and start
again. Not ideal, but safe.

Being able to avoid that by using SSI information was in the back of my
mind, but with no idea how to even begin to tackle it. What you've
mentioned here is helpful and I'd be interested if you could share a bit
more of your experience in the area.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services