Re: I'd like to discuss scaleout at PGCon

From: Simon Riggs
On 1 June 2018 at 16:56, Ashutosh Bapat
<ashutosh(dot)bapat(at)enterprisedb(dot)com> wrote:
>> I think partitioning + FDW provide basic infrastructure for
>> distributing data, planning queries working with such data. We need
>> more glue to support node management, cluster configuration. So, I
>> agree with your statement. But I think it was clear from the
beginning
>> that we need more than FDW and partitioning.
>
> No, it wasn't clear. But I'm glad to hear it. It might actually work
then.

I found a possibly interesting description in the XL manual. Although
XL performs various pushdowns like FDW, XL seems to perform some kind
of joins with datanode-to-datanode communication. Doesn't this prove
that the FDW approach can't handle those joins optimally? What kind
of joins use the shared queue?

https://www.postgres-xl.org/documentation/pg-xc-specifics.html
--------------------------------------------------
shared_queues (integer)
Datanode Only

For some joins that occur in queries, data from one Datanode may need
to be joined with data from another Datanode. Postgres-XL uses shared
queues for this purpose. During execution each Datanode knows if it
needs to produce or consume tuples, or both.

Note that there may be mulitple shared_queues used even for a single
query. So a value should be set taking into account the number of
connections it can accept and expected number of such joins occurring
simultaneously.
--------------------------------------------------

Regards
MauMau