Re: Allowing parallel pg_restore from pipe

On 04/25/2013 12:56 PM, Timothy Garnett wrote:

> As the OP, I'll just note that my organization would definitely find use
> for a parallel migrator tool as long as it supported doing a selection
> of tables (i.e. -t / -T) in addition to the whole database and it
> supported or we were able to patch in an option to cluster as part of
> the migration (the equivalent of something like
> https://github.com/tgarnett/postgres/commit/cc320a71 ).

If you need something like this short term, we actually found a way to
do it ourselves for a migration we performed back in October. The secret
is xargs with the -P option:

xargs -I{} -P 8 -a table-list.txt \
bash -c "pg_dump -Fc -t {} my_db | pg_restore -h remote -d my_db"

Fill table-list.txt with as many, or as few tables as you want. The
above example would give you 8 parallel threads. Well equipped systems
may be able to increase this.

Admittedly it's a gross hack, but it works. :)

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-676-8870
sthomas(at)optionshouse(dot)com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email