pgsql: Add ideas for concurrent pg_dump and pg_restore: < * pg_dump > *

From: momjian(at)postgresql(dot)org (Bruce Momjian)
To: pgsql-committers(at)postgresql(dot)org
Subject: pgsql: Add ideas for concurrent pg_dump and pg_restore: < * pg_dump > *
Date: 2008-03-04 01:33:32
Message-ID: 20080304013332.DC7D1754108@cvs.postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-committers

Log Message:
-----------
Add ideas for concurrent pg_dump and pg_restore:

< * pg_dump
> * pg_dump / pg_restore
> o Allow pg_dump to utilize multiple CPUs and I/O channels by dumping
> multiple objects simultaneously
>
> The difficulty with this is getting multiple dump processes to
> produce a single dump output file.
> http://archives.postgresql.org/pgsql-hackers/2008-02/msg00205.php
>
> o Allow pg_restore to utilize multiple CPUs and I/O channels by
> restoring multiple objects simultaneously
>
> This might require a pg_restore flag to indicate how many
> simultaneous operations should be performed. Only pg_dump's
> -Fc format has the necessary dependency information.
>
> o To better utilize resources, restore data, primary keys, and
> indexes for a single table before restoring the next table
>
> Hopefully this will allow the CPU-I/O load to be more uniform
> for simultaneous restores. The idea is to start data restores
> for several objects, and once the first object is done, to move
> on to its primary keys and indexes. Over time, simultaneous
> data loads and index builds will be running.
>
> o To better utilize resources, allow pg_restore to check foreign
> keys simultaneously, where possible
> o Allow pg_restore to create all indexes of a table
> concurrently, via a single heap scan
>
> This requires a pg_dump -Fc file because that format contains
> the required dependency information.
> http://archives.postgresql.org/pgsql-general/2007-05/msg01274.php
>
> o Allow pg_restore to load different parts of the COPY data
> simultaneously
< single heap scan, and have a restore of a pg_dump somehow use it
> single heap scan, and have pg_restore use it
< http://archives.postgresql.org/pgsql-general/2007-05/msg01274.php

Modified Files:
--------------
pgsql/doc:
TODO (r1.2251 -> r1.2252)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/TODO?r1=1.2251&r2=1.2252)
pgsql/doc/src/FAQ:
TODO.html (r1.760 -> r1.761)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/src/FAQ/TODO.html?r1=1.760&r2=1.761)

Browse pgsql-committers by date

  From Date Subject
Next Message Bruce Momjian 2008-03-04 01:38:37 pgsql: Add URL for: o Allow pg_restore to utilize multiple CPUs
Previous Message User Nanardon 2008-03-04 00:32:45 pgrpm - pgrpm: - no promote epoch in version comparison (otherwise 4.4.8