Re: emergency outage requiring database restart

On Mon, Oct 17, 2016 at 1:39 PM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
> On Thu, Oct 13, 2016 at 4:13 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Merlin Moncure <mmoncure(at)gmail(dot)com> writes:
>>> Today I had an emergency production outage on a server.
>>> ...
>>> Adding all this up it smells like processes were getting stuck on a spinlock.
>>
>> Maybe. If it happens again, probably the most useful debug data would
>> be stack traces from some of the busy processes.
>
> Another odd datapoint on this server. Things were running pretty good
> but an application crashed on a missing view. Trying to recreate the
> view, I got:
>
> CREATE OR REPLACE VIEW vw_ApartmentQueueLastGood AS
> SELECT ...
>
> ERROR: type "vw_apartmentqueuelastgood" already exists
> HINT: A relation has an associated type of the same name, so you must
> use a name that doesn't conflict with any existing type.
>
> ...which was pretty strange. I had to manually delete the pg_type
> record in order to create the view. I'm getting more reports of
> 'could not open relation with oid=X' errors so I could be facing data
> corruption :(.

castaging=# CREATE OR REPLACE VIEW vw_ApartmentSample AS
castaging-# SELECT ...
ERROR: 42809: "pg_cast_oid_index" is an index
LINE 11: FROM ApartmentSample s
^
LOCATION: heap_openrv_extended, heapam.c:1304

should I be restoring from backups?

merlin