Re: Scheduled maintenance affecting gitmaster

2011/2/14 Stefan Kaltenbrunner <Stefan(at)kaltenbrunner(dot)cc>:
> On 02/14/2011 10:09 AM, Magnus Hagander wrote:
>> On Mon, Feb 14, 2011 at 07:13, Stefan Kaltenbrunner
>> <stefan(at)kaltenbrunner(dot)cc> wrote:
>>> On 02/14/2011 01:27 AM, Tom Lane wrote:
>>>>
>>>> Magnus Hagander<magnus(at)hagander(dot)net>  writes:
>>>>>
>>>>> Unfortunately, one of the worst-case scenarios appears to have
>>>>> happened - a machine did not come back up after a reboot.
>>>>> ...
>>>>> We'll get back to you with more information as soon as we have it.
>>>>
>>>> I didn't see any followup to this?
>>>
>>> yeah - the hosting company managed to reboot the box for us which brought it
>>> back to life in the middle of the night (with both magnus and me asleep).
>>
>> Indeed. But the good news is that once it came back up, the VM with
>> the git server started ok :-)
>>
>>
>>>> gitmaster seems to be responding as of now, is it safe to push?
>>>
>>> yes it is - however we will need to schedule another maintenance window soon
>>> to finish the stuff we actually wanted to do.
>>
>> So, after some discussion with Stefan, we (well, I guess I) decided we
>> should just go ahead and declare the maintenance window not closed
>> yet, and finish off the upgrade right now :-) Given that the majority
>> of our commits don't happen now, we'll hopefully have it done by the
>> time the US folks wake up again.
>>
>> So, maintenance window again, starting now, and we'll let you know as
>> soon as we're done. And we're definitely hoping for the machine to
>> come back up properly this time :-)
>
> and it did not... We are trying to figure out what the actual problem
> here really is because it seems to boot just fine when powercycled just
> not with a software initiated reboot.
> We will notify once we have more information...
>

Does it make sense to get some console link or ipmi set up for those
crucial parts of the infrastructure ?

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support