Re: Parallel Sort

On Tue, May 14, 2013 at 11:50 AM, Noah Misch <noah(at)leadboat(dot)com> wrote:
> On Mon, May 13, 2013 at 09:52:43PM +0200, Kohei KaiGai wrote:
>> 2013/5/13 Noah Misch <noah(at)leadboat(dot)com>
>> > The choice of whether to parallelize can probably be made a manner similar
>> > to
>> > the choice to do an external sort: the planner guesses the outcome for
>> > costing
>> > purposes, but the actual decision is made at execution time. The planner
>> > would determine a tuple count cutoff at which parallelism becomes
>> > favorable,
>> > and tuplesort would check that to establish its actual decision.
>>
>> It probably crossovers my problem consciousness to off-load CPU bounds
>> workloads; that I partially tried to implement on writable foreign table
>> feature.
>> Not only sorting stuff, I think it may be worthful to have capability to
>> push
>> heavy workload (like sort, aggregate or complex target-list) out external
>> computing resources.
>> However, I doubt whether the decision to parallelize should be done in
>> execution time, rather than plan stage. For example, in case when we
>> have enough number of records and 10-core multiprocessor, the wise
>> plan may take parallel data load by 10-processors, partial-sort by 10-
>> processors individually, then merge-sort. It needs fundamental different
>> tree structure from the traditional single-processors based plan-tree.
>
> That's taking a few steps more in the direction of parallel general query; at
> some point, the planner would definitely become parallel-aware. For the
> narrower topic of parallel sort, I don't think it's necessary. The node tree
> supplying the sort can't run in parallel (yet), and the node pulling from the
> sort won't receive the first tuple until the sort is complete. For the time
> being, the planner just needs to know enough about the sort node's projected
> use of parallelism to estimate cost.

You know what would be a low-hanging fruit that I've been thinking
would benefit many of my own queries?

"Parallel" sequential scan nodes. Even if there's no real parallelism
involved, when a query has to scan the same table at multiple nodes,
if it's big, it would be worth parallelizing the scans to transform
them into synchro scans.

I have absolutely no idea how this would work easily without forked
workers, because the scans might be buried in more complex execution
trees. But still, it's worth considering, that parallelizing may
benefit more than core usage.

If execution nodes could be paused at arbitrary points, a "parallel
scan" node could pause one branch that has consumed the circular
buffer, letting another branches consume their part, and thus
"parallelizing" branch execution. But this would be perhaps more
complex than simply forking.