pgsql: Add parallel pg_dump option.

From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: pgsql-committers(at)postgresql(dot)org
Subject: pgsql: Add parallel pg_dump option.
Date: 2013-03-24 15:39:44
Message-ID: E1UJn1M-0001uh-Nw@gemulon.postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-committers pgsql-hackers

Add parallel pg_dump option.

New infrastructure is added which creates a set number of workers
(threads on Windows, forked processes on Unix). Jobs are then
handed out to these workers by the master process as needed.
pg_restore is adjusted to use this new infrastructure in place of the
old setup which created a new worker for each step on the fly. Parallel
dumps acquire a snapshot clone in order to stay consistent, if
available.

The parallel option is selected by the -j / --jobs command line
parameter of pg_dump.

Joachim Wieland, lightly editorialized by Andrew Dunstan.

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/9e257a181cc1dc5e19eb5d770ce09cc98f470f5f

Modified Files
--------------
doc/src/sgml/backup.sgml | 18 +
doc/src/sgml/perform.sgml | 9 +
doc/src/sgml/ref/pg_dump.sgml | 89 +++-
src/bin/pg_dump/Makefile | 2 +-
src/bin/pg_dump/compress_io.c | 10 +
src/bin/pg_dump/dumputils.c | 86 ++-
src/bin/pg_dump/dumputils.h | 13 +-
src/bin/pg_dump/parallel.c | 1293 +++++++++++++++++++++++++++++++++
src/bin/pg_dump/parallel.h | 85 +++
src/bin/pg_dump/pg_backup.h | 11 +-
src/bin/pg_dump/pg_backup_archiver.c | 735 +++++++------------
src/bin/pg_dump/pg_backup_archiver.h | 35 +-
src/bin/pg_dump/pg_backup_custom.c | 88 +++-
src/bin/pg_dump/pg_backup_db.c | 20 +-
src/bin/pg_dump/pg_backup_directory.c | 264 +++++++-
src/bin/pg_dump/pg_backup_tar.c | 8 +-
src/bin/pg_dump/pg_dump.c | 681 ++++++++++--------
src/bin/pg_dump/pg_dump.h | 3 +
src/bin/pg_dump/pg_dump_sort.c | 92 +++-
src/bin/pg_dump/pg_dumpall.c | 20 +-
src/bin/pg_dump/pg_restore.c | 17 +-
src/tools/msvc/Mkvcbuild.pm | 5 +
22 files changed, 2765 insertions(+), 819 deletions(-)

Responses

Browse pgsql-committers by date

  From Date Subject
Next Message Tom Lane 2013-03-24 19:35:46 pgsql: Fix some unportable constructs in parallel pg_dump code.
Previous Message Tom Lane 2013-03-23 23:18:10 pgsql: Update time zone abbreviation lists for changes missed since 200

Browse pgsql-hackers by date

  From Date Subject
Next Message Andrew Dunstan 2013-03-24 15:44:34 parallel pg_dump
Previous Message Greg Smith 2013-03-24 14:26:12 Re: Let's invent a function to report lock-wait-blocking PIDs