Re: parallel pg_restore

On Sun, 2008-09-21 at 18:15 -0400, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > I am working on getting parallel pg_restore working. I'm currently
> > getting all the scaffolding working, and hope to have a naive prototype
> > posted within about a week.
>
> > The major question is how to choose the restoration order so as to
> > maximize efficiency both on the server and in reading the archive.
>
> One of the first software design principles I ever learned was to
> separate policy from mechanism. ISTM in this first cut you ought to
> concentrate on mechanism and let the policy just be something dumb
> (but coded separately from the infrastructure). We can refine it after
> that.

Agreed. We musn't make too many built in assumptions about the best way
to parallelise the restore.

For example, running all CREATE INDEX at same time may help I/O on the
scan but it may also swamp memory and force additional I/O as a result.

We might need a setting for total memory available, so pg_restore can
try not to run tasks that will exceed that across settings. Preferably
this wouldn't be just a pg_restore setting.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support