Re: How to estimate the shared memory size required for parallel scan?

On Sun, Aug 19, 2018 at 12:01 AM, Masayuki Takahashi
<masayuki038(at)gmail(dot)com> wrote:
>> It's up to you to design a struct to hold whatever data,
> spinlocks, LWLocks, atomics etc you might need to orchestrate your
> parallel scan.
>
> If FDW(ex. cstore_fdw) does not need to share some information among
> workers more than PostgreSQL core in parallel scan, does it not need
> to allocate DSM?

Right. You don't have to supply InitializeDSMForeignScan,
ReInitializeWorkerForeignScan, InitializeWorkerForeignScan functions.
If you just supply an IsForeignScanParallelSafe function that returns
true, that would allow your FDW to be used inside parallel workers and
wouldn't need any extra shared memory, but it wouldn't be a "parallel
scan". It would just be "parallel safe". Each process that does a
scan of your FDW would expect a full normal scan (presumably returning
the same tuples in each process). That means it can be used, for
example, on the inner side of a join, where the outer side comes from
a parallel scan. Like file_fdw can.

A true parallel scan of an FDW would be one where each process emits
an arbitrary fraction of the tuples, but together they emit all of the
tuples. You'd almost certainly need to use some shared memory to
coordinate that. To say that you support that, I think your
GetForeignPaths() function would need to call add_partial_path(). And
unless I'm mistaken, whether or not InitializeDSMForeignScan etc are
called might be the only indication you get of whether you need to run
in parallel-aware mode. I haven't personally heard of any FDWs that
can do this yet, but I just tried hacking file_fdw to register a
partial path and it seems to work (though of course the results are
duplicated because the emitted tuples are not actually partial).

--
Thomas Munro
http://www.enterprisedb.com