Re: The plan for FDW-based sharding

On 7 March 2016 at 23:02, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Fri, Mar 4, 2016 at 11:17 PM, Craig Ringer <craig(at)2ndquadrant(dot)com>
> wrote:
> > If FDW-based sharding works, I'm happy enough, I have no horse in this
> race.
> > If it doesn't work I don't much care either. What I'm worried about is
> it if
> > works like partitioning using inheritance works - horribly badly, but
> just
> > well enough that it's served as an effective barrier to doing anything
> > better.
> >
> > That's what I want to prevent. Sharding that only-just-works and then
> stops
> > us getting anything better into core.
>
> That's a reasonable worry. Thanks for articulating it so clearly.
> I've thought about that issue and I admit it's both real and serious,
> but I've sort of taken the attitude of saying, well, I don't know how
> to solve that problem, but there's so much other important work that
> needs to be done before we get to the point where that's the blocker
> that solving that problem doesn't seem like the most important thing
> right now.

[snip explanation]

> I think your concern is
> valid, and I share it. But I just fundamentally believe that it's
> better to enhance what we have than to start inventing totally new
> abstractions. The FDW API is *really* powerful, and getting more
> powerful, and I just have a very hard time believing that starting
> over will be better. Somebody can do that if they like and I'm not
> gonna get in the way, but if it's got problems that could have been
> avoided by basing that same work on the FDW stuff we've already got, I
> do plan to point that out.

Yep. As has been noted, each of these improvements is useful in their own
right, and I'm not sure anyone's against them, just
concerned about whether the overall vision for sharding will work out.

Personally I think that once the FDW infrastructure is closer to being
usable for sharding, when we're at the point where new patches are proposed
that're really specifically for sharding and not so general-use FDW
improvements, that's when it'd be well worth building a proof of concept
sharding implementation. Find unexpected wrinkles and issues before
starting to stream stuff into core that can't be easily removed again. That
was certainly useful when building BDR, and even then we still found lots
of things that required revision, often repeatedly.

Either that, or bless experimental features/API as an official concept. I'd
quite like that myself - stuff that's in Pg, but documented as "might
change or go away in the next release, experimental feature". As we're
doing more stuff that spans multiple release cycles, where patches in a
prior cycle might need revision based on what we learn in a later one, we
might need more freedom to change things that're committed and user visible.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services