Re: Support Parallel Query Execution in Executor

"Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote
>
> This is exactly the bit of optimism I was questioning. We've already
> been sweating blood trying to reduce multiprocessor contention on data
> structures in which collisions ought to be avoidable (ie, buffer arrays
> where you hope not everyone is hitting the same buffer at once). I
> think passing large volumes of data between different processes is going
> to incur quite a lot of locking overhead, pipeline stalls for cache line
> transfers, etc, etc, because heavy contention for the transfer buffer is
> simply not going to be avoidable.
>

Yes, there is overhead. Let's say how can we tackle them one by one:

* lock contention of transfer buffer *
A general way is partitioning. We can try to alleviate the lock contention
by sharing transfer buffer only between each two processes (master and
slave).

* large amount of data *
Thinking of the simplest master-slave seqscan senario, I am copying the
HeapTuple so far (like the code in sort read/write tuple), but potentially
we can only pass the offset and pinned page, which will require the slave
process pinned the page for a little longer time, say every 5 pages. I
didn't see any application can cause Postgres pinned a lot pages, so this
might be one step to reduce the data volumn.

On the other hand, why do we need parallel execution anyway? Because if we
got a 4 way SMP with strong IO device, we may want to trade the performance
with more resources for some queries.

Regards,
Qingqing