Re: Horizontal scalability/sharding

On Tue, Sep 1, 2015 at 4:15 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> On 2015-08-31 20:54:51 -0400, Bruce Momjian wrote:
>> Uh, we already have a list of things we need to add to FDWs to make them
>> work, and Citus Data has provided a document of more things that are
>> needed, https://goo.gl/vJWF85. I am not sure how much bigger a red flag
>> you want to confirm that everyone agrees that major FDW improvements are
>> a requirement for this.
>
> Several people saying that the FDW infrastructure isn't sufficient right
> now is pretty far from implying that all of them agree that the FDW API
> is the way to go.
>
> I'm not sure myself. If it works out it's going to save us some work and
> make it more realistic to get there sometime not too far off. But I'm
> afraid that the resulting system will feel like our current partitioning
> implemenentation. Yes, it kinda works, but it's hard to get started, it
> doesn't support too many features and you're kind afraid your relatives
> will see what you've done.

I'm not averse to making the "connect to the remote nodes" part of
this solution use something other than the FDW infrastructure at some
point in time if somebody's prepared to build something better. On
the other hand, I think it's extremely clear that the FDW
infrastructure has a large amount of potential upon which we have
thoroughly failed to capitalize. Patches have already been written
for UPDATE/DELETE pushdown and for join pushdown. Those patches have
been around for some time, but progress has been slow. Core
infrastructure exists to allow sort pushdown, but nobody's done
anything with it. Aggregate pushdown hasn't happened yet due to the
dependency on upper planner path-ification, but it's not as if some
alternative to the FDW interface is going to dodge that problem.

It would be a bad idea to cling blindly to the FDW infrastructure if
it's fundamentally inadequate to do what we want. On the other hand,
it would also be a bad idea to set about recreating it without a
really good reason, and - just to take one example - the fact that it
doesn't currently push down DML operations to the remote side is not a
really good reason to rewrite the whole thing. On the contrary, it's
a reason to put some energy into the already-written patch which
implements that optimization.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company