Re: Horizontal scalability/sharding

On Mon, Sep 7, 2015 at 1:33 PM, Ahsan Hadi <ahsan(dot)hadi(at)enterprisedb(dot)com> wrote:
> I
>
> On Monday, September 7, 2015, Ashutosh Bapat
> <ashutosh(dot)bapat(at)enterprisedb(dot)com> wrote:
>>
>>
>>
>> On Sat, Sep 5, 2015 at 4:22 AM, Ozgun Erdogan <ozgun(at)citusdata(dot)com> wrote:
>>>
>>> Hey Robert,
>>>
>>>> Now the question is, where should the code that does all of this live?
>>>> postgres_fdw? Some new, sharding-specific FDW? In core? I don't
>>>> know for sure, but what I do know is that we could make a lot of
>>>> progress over where we are today by just improving postgres_fdw, and I
>>>> don't think those improvements are even all that difficult. If we
>>>> decide we need to implement something new, it's going to be a huge
>>>> project that will take years to complete, with uncertain results. I'd
>>>> rather have a postgres_fdw-based implementation that is imperfect and
>>>> can't handle some kinds of queries in 9.6 than a promise that by 9.9
>>>> we'll have something really great that handles MPP perfectly.
>>>
>>>
>>> Distributed shuffles (Map/Reduce) are hard. When we looked at using FDWs
>>> for pg_shard, we thought that Map/Reduce would require a comprehensive
>>> revamp of the APIs.
>>>
>>> For Citus, a second part of the question is as FDW writers. We
>>> implemented cstore_fdw, json_fdw, and mongo_fdw, and these wrappers don't
>>> benefit from even the simple join pushdown that doesn't require Map/Reduce.
>>
>>
>> I didn't get this. Join pushdown infrastructure (chiefly set of hooks
>> provided in join planning paths) is part of 9.5. Isn't that sufficient to
>> implement join push-down for above FDWs? Or FDW writers are facing problems
>> while implementing those hooks. In either case that should be reported on
>> hackers.
>>
>
>
> I don't think any FDW writer (other the postgres_fdw) has tried to implement
> join push down in the respective FDW's using the new API.

Well, 'jdbc_fdw2' seems to implement deparsing at some level:
https://github.com/heimir-sverrisson/jdbc2_fdw/blob/master/deparse.c

...but this was likely a copy/paste job from the postgres_fdw. It
should not escape note that the deparsing strategy has dependencies on
the optimizer and the parser. This is not good; asking the FDW
implementations to implement SQL optimizations is not a sustainable
solution. They should be indicating, for example, "I support ANSI-92
SQL", and the postgres planner should be deparsing the foreign table
definition and rewriting it, not the other way around.

merlin