Re: The plan for FDW-based sharding

Bruce Momjian wrote:
> On Wed, Feb 24, 2016 at 01:08:29AM +0000, Simon Riggs wrote:

> > It's never been our policy to try to include major projects in single code
> > drops. Any move of XL/XC code into PostgreSQL core would need to be done piece
> > by piece across many releases. XL is definitely too big for the elephant to eat
> > in one mouthful.
>
> Is there any plan to move the XL/XC code into Postgres? If so, I have
> not heard of it. I thought everyone agreed it was too much code change,
> which is why it is a separate code tree. Is that incorrect?

Yes, I think that's incorrect.

What was said, as I understood it, is that Postgres-XL is too big to
merge in a single commit -- just like merging BDR would have been.
Indulge me while I make a parallel with BDR for a bit.
2ndQuadrant never pushed for merging BDR in a single commit; what was
done was to split it, and propose individual pieces for commit. Many of
these pieces are now already committed (event triggers, background
workers, logical decoding, replication slots, and many others). The
"BDR patch" is now much smaller, and it's quite possible that we will
see it merged someday. Will it be different from what it was when the
BDR project started, all those years ago? You bet. Having the
prototype BDR initially was what allowed the whole plan to make sense,
because it showed that the pieces interacted in the right ways to make
it work as a whole.

(I'm not saying 2ndQuadrant is so wise to do things this way. I'm
pretty sure you can see the same thing in parallel query development,
for instance.)

In the same way, Postgres-XL is far too big to merge in a single commit.
But that doesn't mean it will never be merged. What is more likely to
happen instead is that some pieces of it are going to be submitted
separately for consideration. It is a slow process, but progress is
real and tangible. We know this process will yield a useful outcome,
because the architecture has already proven by the existance of
Postgres-XL itself. It's the prototype that proves the overall design,
even if the pieces change shape during the process. (Really, it's way
more than merely a prototype at this point because of how long it has
matured.)

In contrast, we don't have a prototype for FDW-based sharding; as you
admitted, there is no actual plan, other than "let's push FDWs in this
direction and hope that sharding will emerge". We don't really know
what pieces we need or how will they interact with each other; we have a
vague idea of a direction but there's no clear path forward. As the
saying goes, if you don't know where you're going, you will probably end
up somewhere else.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services