parallel pg_restore

From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: parallel pg_restore
Date: 2008-09-21 19:28:54
Message-ID: 48D6A076.90107@dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

I am working on getting parallel pg_restore working. I'm currently
getting all the scaffolding working, and hope to have a naive prototype
posted within about a week.

The major question is how to choose the restoration order so as to
maximize efficiency both on the server and in reading the archive. My
thoughts are currently running something like this:

* when an item is completed, reduce the dependency count for each
item that depends on it by 1.
* when an item has a dependency count of 0 it is available for
execution, and gets moved to the head of the queue.
* when a new worker spot becomes available, if there not currently a
data load running then pick the first available data load,
otherwise pick the first available item.

This would mean that loading a table would probably be immediately
followed by creation of its indexes, including PK and UNIQUE
constraints, thus taking possible advantage of synchronised scans, data
in file system buffers, etc.

Another question is what we should do if the user supplies an explicit
order with --use-list. I'm inclined to say we should stick strictly with
the supplied order. Or maybe that should be an option.

Thoughts and comments welcome.

cheers

andrew

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2008-09-21 19:57:22 Re: Predictable order of SQL commands in pg_dump
Previous Message Dmitry Koterov 2008-09-21 19:22:20 Re: Predictable order of SQL commands in pg_dump