Re: Jira database won't start after disk filled up

On Fri, Mar 2, 2018 at 4:32 PM, Paul Costello <paulc1217(at)gmail(dot)com> wrote:

> I have a database that wouldn't start due to the disk filling up back on
> 1/10, unbeknownst to us until 2/27. This is jira, so it's critical data.
> It appears jira was running in memory that entire time.
>

Those first two sentences seem contradictory...

>
> I needed to run pg_resetxlog -f in order to start the database. It
> started, but upon logging in I found the system catalog and some data to be
> corrupt.
>

Once you did this, fixing the data is really on you. Postgres has no way to
know what any of the data mean, nor how to decide what to keep and what to
toss on those conflicting rows with duplicate keys.

What I'd personally do is take your 1/5 backup, then merge in rows for
tickets and affiliated data from whatever you can recover in the current
database copy you have. Once that's done, run jira's built-in integrity
checker then do a full export to XML backup format. Finally re-import that
into a fresh jira so you know what's in there is consistent. You'll
probably also have to cross-reference the attachments directory for missing
tickets and clean up those files (or synthesize tickets for them).

If your jira is configured to send email somewhere on ticket updates,
gathering those (even if it is in multiple people's mailboxes) and
recreating ticket info from them would also move you along.

You will lose some of your data because not all of it was written to disk.