Bringing up new slaves faster

From: François Deliège <fdeliege(at)gmail(dot)com>
To: pgsql-performance(at)postgresql(dot)org
Subject: Bringing up new slaves faster
Date: 2013-09-24 18:52:58
Message-ID: CAPMnNqdwheZZ=d939MueS-i50XUUpO9YbH=qU-LrE=SeQ02R6Q@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Hi all,

Bringing up new slaves is a task we are performing very frequently. Our
current process is:
1. launch a new instance, perform a pg_basebackup
2. start postgresql on the slave and let the slave recover from the archive
all the xlog files it missed and that got created during step 1.
3. let the slave stream from the master

We use pg_basebackup with the -X option to stream the wal files created and
get rid of step 2.

However, when the postgresql server starts on the slave, it first tries to
recover the files from our archive even if the files have been copied to
the pg_xlog directory. Recovering from the archive is slow and seem to be
unnecessary given that the xlogs have been copied.

To address that problem we wrote a little shell script that changes that
order. The script let the slave make a first attempt at using the local
xlog file if it exists, and falls back to recovering from the archive in
case of failure.

While this seems to be a common use case, we could not find much info about
this.
- Are we missing something?
- What are the trade-offs of such approach?
- Is there any additional risks of bringing up new slave with that recovery
option?

Thanks a lot!

Francois

Browse pgsql-performance by date

  From Date Subject
Next Message Merlin Moncure 2013-09-24 21:49:10 Re: Slow plan for MAX/MIN or LIMIT 1?
Previous Message didier 2013-09-24 18:03:20 Re: Planner performance extremely affected by an hanging transaction (20-30 times)?