Re: Parallel Sort

On 13 May 2013 15:57, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> I think you need to think in terms of restricting the problem space

+1

> One obvious suggestion is to forbid the workers from
> doing any database access of their own at all --- the parent would
> have to do any required catalog lookups for sort functions etc.

+1

> I think we should also seriously think about relying on fork() and
> copy-on-write semantics to launch worker subprocesses, instead of
> explicitly copying so much state over to them. Yes, this would
> foreclose ever having parallel query on Windows, but that's okay
> with me (hm, now where did I put my asbestos longjohns ...)

If we relied on some kind of inherited state we could easily make the
mistake of relying on something that isn't actually being maintained
correctly in the worker. Luckily (?) that is exactly the stance we
need to make this work on Windows. Other than that, releasing on
Windows in later release sounds sensible, otherwise we'll just delay
the development so much it will still happen in the "later" timeframe,
just the chance of an earlier release on Linux/BSD will be missed.

For example, the idea of managing separate subtransactions in each
worker sounds nuts. Impressive, if you're already thinking about
parallel DML that can self recover halfway through a statement and
then continue processing, but that's a little advanced. The way to
think about this is as a 10 year journey, not as a single feature.

-1 for forking

> Both of these lines of thought suggest that the workers should *not*
> be full-fledged backends.

+1 to the idea of workers != masters

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services