From: | Etsuro Fujita <etsuro(dot)fujita(at)gmail(dot)com> |
---|---|
To: | Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com> |
Cc: | Justin Pryzby <pryzby(at)telsasoft(dot)com>, Andrey Lepikhov <a(dot)lepikhov(at)postgrespro(dot)ru>, "movead(dot)li" <movead(dot)li(at)highgo(dot)ca>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
Subject: | Re: Asynchronous Append on postgres_fdw nodes. |
Date: | 2021-02-01 03:06:09 |
Message-ID: | CAPmGK14xEcDpEpA+iMgNkTePPHmo9k1qXMmYzP1xyKSk61d4rw@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Tue, Nov 17, 2020 at 6:56 PM Etsuro Fujita <etsuro(dot)fujita(at)gmail(dot)com> wrote:
> * I haven't yet done anything about the issue on postgres_fdw's
> handling of concurrent data fetches by multiple ForeignScan nodes
> (below *different* Append nodes in the query) using the same
> connection discussed in [2]. I modified the patch to just disable
> applying this feature to problematic test cases in the postgres_fdw
> regression tests, by a new GUC enable_async_append.
A solution for the issue would be a scheduler designed to handle such
data fetches more efficiently, but I don’t think it’s easy to create
such a scheduler. Rather than doing so, I'd like to propose to allow
FDWs to disable async execution of them in problematic cases by
themselves during executor startup in the first cut. What I have in
mind for that is:
1) For an FDW that has async-capable ForeignScan(s), we allow the FDW
to record, for each of the async-capable and non-async-capable
ForeignScan(s), the information on a connection to be used for the
ForeignScan into EState during BeginForeignScan().
2) After doing ExecProcNode() to each SubPlan and the main query tree
in InitPlan(), we give the FDW a chance to a) reconsider, for each of
the async-capable ForeignScan(s), whether the ForeignScan can be
executed asynchronously as planned, based on the information stored
into EState in #1, and then b) disable async execution of the
ForeignScan if not.
#1 and #2 would be done after initial partition pruning, so more
async-capable ForeignScans would be executed asynchronously, if other
async-capable ForeignScans conflicting with them are removed by that
pruning.
This wouldn’t prevent us from adding a feature like what was proposed
by Horiguchi-san later.
BTW: while considering this, I noticed some bugs with
ExecAppendAsyncBegin() in the previous patch. Attached is a new
version of the patch fixing them. I also tweaked some comments a
little bit.
Best regards,
Etsuro Fujita
Attachment | Content-Type | Size |
---|---|---|
async-wip-2021-02-01.patch | application/octet-stream | 80.5 KB |
From | Date | Subject | |
---|---|---|---|
Next Message | Masahiko Sawada | 2021-02-01 03:09:12 | Re: avoid bitmapOR-ing indexes with scan condition inconsistent with partition constraint |
Previous Message | Amit Kapila | 2021-02-01 02:54:37 | Re: Single transaction in the tablesync worker? |