Re: The plan for FDW-based sharding

Hi,

On 03/01/2016 08:02 PM, Bruce Momjian wrote:
> On Tue, Mar 1, 2016 at 07:56:58PM +0100, Petr Jelinek wrote:
>> Note that I am not saying that other discussed approaches are any
>> better, I am saying that we should know approximately what we
>> actually want and not just beat FDWs with a hammer and hope sharding
>> will eventually emerge and call that the plan.
>
> I will say it again --- FDWs are the only sharding method I can think
> of that has a chance of being accepted into Postgres core.

I don't quite see why that would be the case. Firstly, it assumes that
FDW-based approach is going to work, but given the lack of prototype or
even a technical analysis discussing the missing pieces, that's very
difficult to judge.

I find it a bit annoying that there are objections from people who
implemented (or attempted to implement) sharding on PostgreSQL, yet no
reasonable analysis of their arguments and how the FDW approach will
address them. My my understanding is they deem FDWs a bad foundation for
sharding because it was designed for a different purpose, but the
abstractions are a bad fit for sharding (which assumes isolated nodes,
certain form of execution etc.)

> It is a plan, and if it fails, it fails. If is succeeds, that's
> good. What more do you want me to say? I know of no other way to
> answer the questions you asked above.

Well, wouldn't it be great if we could do the decision based on some
facts and not mere belief that it'll help. That's exactly what Petr is
talking about - the fear that we'll spend a few years working on
sharding based on FDWs, only to find out that it does not work too well.
That'd be a pretty bad outcome, wouldn't it?

My other worry is that we'll eventually mess the FDW infrastructure,
making it harder to use for the original purpose. Granted, most of the
improvements proposed so far look sane and useful for FDWs in general,
but sooner or later that ceases to be the case - there sill be changes
needed merely for the sharding. Those will be tough decisions.

While I disagree with Simon on various things, I absolutely understand
why he was asking about a prototype, and some sort of analysis of what
usecases we expect to support initially/later/never, and what pieces are
missing to get the sharding working. IIRC at the FOSDEM Dev Meeting
you've claimed you're essentially working on a prototype - once we have
the missing FDW pieces, we'll know if it works. I disagree that - it's
not a prototype if it takes several years to find the outcome.

Also, in another branch of this thread you've said this (I don't want to
sprinkle the thread with responses, so I'll just respond here):

> In a way, I don't see any need for an FDW sharding prototype
> because, as I said, we already know XC/XL work, so copying what they
> do doesn't help. What we need to know is if we can get near the XC/XL
> benchmarks with an acceptable addition of code, which is what I
> thought I already said. Perhaps this can be done with FDWs, or some
> other approach I have not heard of yet.

I don't quite understand the reasoning presented here. The XC/XL are not
based on FDWs at all, therefore the need for prototype of the FDW-based
sharding is entirely independent to the fact that these solutions seem
to work quite well.

regards

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services