Re: Parallel query execution

On Tuesday, January 15, 2013, Gavin Flower wrote:

> On 16/01/13 11:14, Bruce Momjian wrote:
>
> I mentioned last year that I wanted to start working on parallelism:
>
> https://wiki.postgresql.org/wiki/Parallel_Query_Execution
>
> Years ago I added thread-safety to libpq. Recently I added two parallel
> execution paths to pg_upgrade. The first parallel path allows execution
> of external binaries pg_dump and psql (to restore). The second parallel
> path does copy/link by calling fork/thread-safe C functions. I was able
> to do each in 2-3 days.
>
> I believe it is time to start adding parallel execution to the backend.
> We already have some parallelism in the backend:
> effective_io_concurrency and helper processes. I think it is time we
> start to consider additional options.
>
> Parallelism isn't going to help all queries, in fact it might be just a
> small subset, but it will be the larger queries. The pg_upgrade
> parallelism only helps clusters with multiple databases or tablespaces,
> but the improvements are significant.
>
> I have summarized my ideas by updating our Parallel Query Execution wiki
> page:
>
> https://wiki.postgresql.org/wiki/Parallel_Query_Execution
>
> Please consider updating the page yourself or posting your ideas to this
> thread. Thanks.
>
>
> Hmm...
>
> How about being aware of multiple spindles - so if the requested data
> covers multiple spindles, then data could be extracted in parallel. This
> may, or may not, involve multiple I/O channels?
>

effective_io_concurrency does this for bitmap scans. I thought there was a
patch in the commitfest to extend this to ordinary index scans, but now I
can't find it. But it still doesn't give you CPU parallelism. The nice
thing about CPU parallelism is that it usually brings some amount of IO
parallelism for free, while the reverse less likely to be so.

Cheers,

Jeff

>