Re: Horizontal scalability/sharding

I

On Monday, September 7, 2015, Ashutosh Bapat <
ashutosh(dot)bapat(at)enterprisedb(dot)com> wrote:

>
>
> On Sat, Sep 5, 2015 at 4:22 AM, Ozgun Erdogan <ozgun(at)citusdata(dot)com
> <javascript:_e(%7B%7D,'cvml','ozgun(at)citusdata(dot)com');>> wrote:
>
>> Hey Robert,
>>
>> Now the question is, where should the code that does all of this live?
>>> postgres_fdw? Some new, sharding-specific FDW? In core? I don't
>>> know for sure, but what I do know is that we could make a lot of
>>> progress over where we are today by just improving postgres_fdw, and I
>>> don't think those improvements are even all that difficult. If we
>>> decide we need to implement something new, it's going to be a huge
>>> project that will take years to complete, with uncertain results. I'd
>>> rather have a postgres_fdw-based implementation that is imperfect and
>>> can't handle some kinds of queries in 9.6 than a promise that by 9.9
>>> we'll have something really great that handles MPP perfectly.
>>>
>>
>> Distributed shuffles (Map/Reduce) are hard. When we looked at using FDWs
>> for pg_shard, we thought that Map/Reduce would require a comprehensive
>> revamp of the APIs.
>>
>> For Citus, a second part of the question is as FDW writers. We
>> implemented cstore_fdw, json_fdw, and mongo_fdw, and these wrappers don't
>> benefit from even the simple join pushdown that doesn't require Map/Reduce.
>>
>
> I didn't get this. Join pushdown infrastructure (chiefly set of hooks
> provided in join planning paths) is part of 9.5. Isn't that sufficient to
> implement join push-down for above FDWs? Or FDW writers are facing problems
> while implementing those hooks. In either case that should be reported on
> hackers.
>
>

I don't think any FDW writer (other the postgres_fdw) has tried to
implement join push down in the respective FDW's using the new API.

>
>> The PostgreSQL wiki lists 85 foreign data wrappers, and only 18 of these
>> have support for joins:
>> https://wiki.postgresql.org/wiki/Foreign_data_wrappers
>>
>> Best,
>> Ozgun
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>

--
Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.