Re: parallel pg_restore

Simon Riggs wrote:
> On Mon, 2008-09-22 at 09:53 +0200, Dimitri Fontaine wrote:
>
>
>>> My intention is to have single-thread restore remain the default, at
>>> least for this go round, and have the user be able to choose
>>> --multi-thread=nn to specify the number of concurrent connections to use.
>>>
>> What about the make famous -j option?
>>
>> -j [jobs], --jobs[=jobs]
>> Specifies the number of jobs (commands) to run simultaneously. If
>> there is more than one -j option, the last one is effective. If
>> the -j option is given without an argument, make will not limit
>> the number of jobs that can run simultaneously.
>>
>
> +1
>
>

If that's the preferred name I have no problem. I'm not sure about the
default argument part, though.

First, I'm not sure out getopt infrastructure actually provides for
optional arguments, and I am not going to remove it in pg_restore to get
around such a problem, at least now.

More importantly, I'm not convinced it's a good idea. It seems more like
a footgun that will potentially try to launch thousands of simultaneous
restore connections. I should have thought that optimal performance
would be reached at some small multiple (say maybe 2?) of the number of
CPUs on the server. You could achieve unlimited parallelism by saying
something like --jobs=99999, but I'd rather that were done very
explicitly instead of as the default value of the parameter.

cheers

andrew