Re: Jira database won't start after disk filled up

Yes, contradictory. We recently disabled email forwarding, so on 1/10 when
the disk filled up, we never received any alerts. New position, so I was
completely unaware of this database, except on a conceptual level, until
Tuesday.

I think the best I can do is get this database back to 1/10. In my first
restore attempt I noticed that the most recent jiraissue.resolutiondate was
1/10, so jira was running in memory the entire time and nothing was flushed
to the db - probably because it was corrupted, and possibly other reasons
related to neglect.

My hope is that I can get the db back to 1/10 and maybe we can, with
Atlassian's help, somehow sync the lucene files back to the db. I don't
think I will have any postgres data to work with beyond 1/10.

Does this still sound do-able with that kind of data gap?

On Fri, Mar 2, 2018 at 3:44 PM, Vick Khera <vivek(at)khera(dot)org> wrote:

> On Fri, Mar 2, 2018 at 4:32 PM, Paul Costello <paulc1217(at)gmail(dot)com> wrote:
>
>> I have a database that wouldn't start due to the disk filling up back on
>> 1/10, unbeknownst to us until 2/27. This is jira, so it's critical data.
>> It appears jira was running in memory that entire time.
>>
>
>
> Those first two sentences seem contradictory...
>
>
>>
>> I needed to run pg_resetxlog -f in order to start the database. It
>> started, but upon logging in I found the system catalog and some data to be
>> corrupt.
>>
>
>
> Once you did this, fixing the data is really on you. Postgres has no way
> to know what any of the data mean, nor how to decide what to keep and what
> to toss on those conflicting rows with duplicate keys.
>
> What I'd personally do is take your 1/5 backup, then merge in rows for
> tickets and affiliated data from whatever you can recover in the current
> database copy you have. Once that's done, run jira's built-in integrity
> checker then do a full export to XML backup format. Finally re-import that
> into a fresh jira so you know what's in there is consistent. You'll
> probably also have to cross-reference the attachments directory for missing
> tickets and clean up those files (or synthesize tickets for them).
>
> If your jira is configured to send email somewhere on ticket updates,
> gathering those (even if it is in multiple people's mailboxes) and
> recreating ticket info from them would also move you along.
>
> You will lose some of your data because not all of it was written to disk.
>
>