Re: Horizontal scalability/sharding

On Sat, Sep 5, 2015 at 4:22 AM, Ozgun Erdogan <ozgun(at)citusdata(dot)com> wrote:

> Hey Robert,
>
> Now the question is, where should the code that does all of this live?
>> postgres_fdw? Some new, sharding-specific FDW? In core? I don't
>> know for sure, but what I do know is that we could make a lot of
>> progress over where we are today by just improving postgres_fdw, and I
>> don't think those improvements are even all that difficult. If we
>> decide we need to implement something new, it's going to be a huge
>> project that will take years to complete, with uncertain results. I'd
>> rather have a postgres_fdw-based implementation that is imperfect and
>> can't handle some kinds of queries in 9.6 than a promise that by 9.9
>> we'll have something really great that handles MPP perfectly.
>>
>
> Distributed shuffles (Map/Reduce) are hard. When we looked at using FDWs
> for pg_shard, we thought that Map/Reduce would require a comprehensive
> revamp of the APIs.
>
> For Citus, a second part of the question is as FDW writers. We implemented
> cstore_fdw, json_fdw, and mongo_fdw, and these wrappers don't benefit from
> even the simple join pushdown that doesn't require Map/Reduce.
>

I didn't get this. Join pushdown infrastructure (chiefly set of hooks
provided in join planning paths) is part of 9.5. Isn't that sufficient to
implement join push-down for above FDWs? Or FDW writers are facing problems
while implementing those hooks. In either case that should be reported on
hackers.

>
> The PostgreSQL wiki lists 85 foreign data wrappers, and only 18 of these
> have support for joins:
> https://wiki.postgresql.org/wiki/Foreign_data_wrappers
>
> Best,
> Ozgun
>

--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company