Make broken database start

From: Дмитрий Шалашов <skaurus(at)gmail(dot)com>
To: pgsql-admin(at)postgresql(dot)org
Subject: Make broken database start
Date: 2014-11-26 13:38:41
Message-ID: CAKPeCUF8HGRyjxc-0ZoYEBg_16Ts1+LYyHLk0bHMS9vGMC0waw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

Hi everyone,

(sorry if this is a dup - I'm not sure if my previous message get through)

I need an advice in database recovery. Postgresql 9.2.9.
Last night we moved one of our databases to another server following this
procedure: https://wiki.postgresql.org/wiki/Streaming_Replication
Replication, slave becomes master, pgbouncer restarts with new config,
everyone happy.
Until soon we noticed strange errors. Turns out data was broken somewhere -
we had unique indexes with duplicate records, we had missing chunks in
toast tables, and the like.
Scale of the problems wasn't clear and I fixed some of the most obvious
problems - recreated indices, cleared duplicates. Around 0:30 I started
database-wide VACUUM, it started printing messages about fixing things.
Everything seemed to work normally.
Around 3:30 database stopped responding to queries; it was noticed around
9:30 and I attempted to stop VACUUM (ctrl+c in psql command line). 30
minutes and nothing. I decided to restart postmaster. After another 30
minutes we switched back to old server. Already more than 6 hours passed
and new server postmaster is still shutting down. And I would like to get
it back to restore yesterday data.

At this point I could do with new server anything that will get database up
and running sooner. kill -9, start, wait for recovery? Server reboot?

Any help will be greatly appreciated.

Best regards,
Dmitriy Shalashov

Browse pgsql-admin by date

  From Date Subject
Next Message John R Pierce 2014-11-26 17:20:20 Re: Active/Active clustering in postgres
Previous Message Jan Lentfer 2014-11-26 11:00:22 Re: Active/Active clustering in postgres