Re: replication behind high lag

On Mon, Mar 25, 2013 at 12:55 PM, AI Rumman <rummandba(at)gmail(dot)com> wrote:
>
>
> On Mon, Mar 25, 2013 at 3:52 PM, Lonni J Friedman <netllama(at)gmail(dot)com>
> wrote:
>>
>> On Mon, Mar 25, 2013 at 12:43 PM, AI Rumman <rummandba(at)gmail(dot)com> wrote:
>> >
>> >
>> > On Mon, Mar 25, 2013 at 3:40 PM, Lonni J Friedman <netllama(at)gmail(dot)com>
>> > wrote:
>> >>
>> >> On Mon, Mar 25, 2013 at 12:37 PM, AI Rumman <rummandba(at)gmail(dot)com>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > I have two 9.2 databases running with hot_standby replication. Today
>> >> > when I
>> >> > was checking, I found that replication has not been working since Mar
>> >> > 1st.
>> >> > There was a large database restored in master on that day and I
>> >> > believe
>> >> > after that the lag went higher.
>> >> >
>> >> > SELECT pg_xlog_location_diff(pg_current_xlog_location(), '0/0') AS
>> >> > offset
>> >> >
>> >> > 431326108320
>> >> >
>> >> > SELECT pg_xlog_location_diff(pg_last_xlog_receive_location(), '0/0')
>> >> > AS
>> >> > receive, pg_xlog_location_diff(pg_last_xlog_replay_location(),
>> >> > '0/0')
>> >> > AS replay
>> >> >
>> >> > receive | replay
>> >> > --------------+--------------
>> >> > 245987541312 | 245987534032
>> >> > (1 row)
>> >> >
>> >> > I checked the pg_xlog in both the server. In Slave the last xlog file
>> >> > -rw------- 1 postgres postgres 16777216 Mar 1 06:02
>> >> > 00000001000000390000007F
>> >> >
>> >> > In Master, the first xlog file is
>> >> > -rw------- 1 postgres postgres 16777216 Mar 1 04:45
>> >> > 00000001000000390000005E
>> >> >
>> >> >
>> >> > Is there any way I could sync the slave in quick process?
>> >>
>> >> generate a new base backup, and seed the slave with it.
>> >
>> >
>> > OK. I am getting these error in slave:
>> > LOG: invalid contrecord length 284 in log file 57, segment 127, offset
>> > 0
>> >
>> > What is the actual reason?
>>
>> Corruption? What were you doing when you saw the error?
>
>
> I did not have enough idea about these stuffs. I got the database now and
> saw the error.
> Is there any way to recover from this state. The master database is a large
> database of 500 GB.

generate a new base backup, and seed the slave with it. if the error
persists, then i'd guess that your master is corrupted, and then
you've got huge problems.