Re: bugs and bug tracking

Nathan,

* Nathan Wagner (nw+pg(at)hydaspes(dot)if(dot)org) wrote:
> So, in order to do some clean up and see how my pgbugs page
> (https://granicus.if.org/pgbugs/) might actually work I've been going
> through bugs and marking their status. A lot of questions arise.

Thanks for working on this!

> A lot of the reports aren't bugs at all, but requests for help. My
> guess is that the users either don't know where to ask or don't
> understand the difference between a bug and not knowing how to do what
> they want to do. Perhaps a more thorough explaination on the submission
> form would be useful.

While I agree with you that they are requests for help, I doubt changing
the submission form would really help much.

[...]

> I have lots of different types of 'not a bug'.

debbugs has categories for more-or-less all of these:

For the case where it's a feature rather than a bug, there's "wishlist".

> Not a bug, the use should have posted to a different list. (e.g. 13602)
> Not a bug, probably user error, which is similar to the above.
> Not a bug, but a bug in the libraries we use (e.g. openssl, 10184)

The ones above would all simply be closed with a comment to the user
about what the issue is.

> Not a bug, works as intended, i.e. the user didn't make a mistake, but
> had an incorrect notion of what it was supposed to do. (11596)

This could be either closed or, if it pops up enough, left as 'wontfix',
so other users don't report it again.

> Not a bug, but the user never got a reply. That is, I decided
> personally that this wasn't actually a bug. (13367)

One thing with debbugs, at least, is that the user gets an initial reply
saying "we got it" and then another whenever the status changes
(including if it gets closed out as not-a-bug or similar).

> And bug 1000 is not a bug, system test.

Eh, not sure we need to worry about that one too much. :)

> Do we care about the difference between any of these? I track them
> differently in my update script, but they all get the same status in the
> db.

I like the set of categories which debbugs provides.

> Can a bug be 'fixed' if there's no actually identifiable commit that
> fixes the bug? See 13516, which Tom Lane claims was fixed in 9.1. A
> grep for 13516 of the git log for both master and REL9_1_STABLE don't
> turn up anything.

Yes, that can certainly happen. It'd be better if the commit which
actually fixed it was found but that's just being idealistic- whatever
we use shouldn't force us to close a bug with a commit (debbugs
certainly doesn't).

> I can't as yet figure out how to match up git commit messages to
> identify every branch in which a fix was applied. I could of course
> load all of the commit messages into a table and compare them that way.

Can't this be done by simply looking for the bug# in the commit log for
each branch and, when found, associating that branch with the bug#?

> Should I mark as "open" (i.e. not "new) any report that has a response?
> More than one response? That would at least distinguish between bug
> reports that at least someone, in theory, has taken a look at, and those
> that haven't been addressed at all.

I'm not sure that distinction is particularly useful, but I know some
systems do it.

> I have created the following statuses:
>
> Fixed - bug is presumably fixed

Ok.

> Unreproduceable - we can't make the system demonstrate this error

This should be a flag or attribute of the bug, not a final disposition.

> Timed Out - the reporter was asked to provide more information and
> didn't respond for a while. I would probably suggest somewhere around a
> month for this. Should there be a 'waiting on feedback' to mark the
> 'pre timed out' phase?

Not sure we need this, isn't it just 'closed'?

> Stale 5281 - the bug hasn't had any activity for >2 years, so just
> close it. If people want to troll through these to give them a better
> status, that would probably be good, but there's still a thousand open
> bugs newer than that.

How is this different from 'timed out'?

> Not Our Bug - looks like a bug, but it's not a bug in postgres. What
> exactly are our bugs? Just what you'd get out of the release tarballs
> or the git repo? Or is there more?

This would also be 'closed', but with a note saying it's not our issue.

> Won't Fix - this is arguably a bug, but for some reason we're not going
> to fix it. Perhaps we intentionally violate the standard, or it's a bug
> against a version we don't support, or we're not going to backpatch it.

I'm not sure that it's actually a bug, it's more of a placeholder that
says "yes, we understand people might think this behavior should be
different, but we don't agree and aren't going to change it."

> Open - this seems to be a bug, or at least we're talking about it and
> it's not where we want to close it. Note of course that "closing" a bug
> just means it will show up somewhere else in my tracker, obviously it
> doesn't affect the mailing list at all.

Yes, closed bugs should still be available for review.

> New - this hasn't been looked at enough for someone to change the status
> to something better.

Alright.

We should also have 'wishlist', imv, as discussed.

> I don't have a "reopened" status. I'm not sure what it means, other
> than it used to be closed, but someone changed it back to open. I don't
> immediately see why we would want to distinguish between this and a
> regular open bug, other than perhaps as a way of conflating status with
> priority. It's easy to make one though if people really want it. I
> probably have too many statuses already.

I think we want to track that it was closed and then opened again but I
don't think it needs a formal status. Generally speaking, we should be
tracking all actions against a given bug (as debbugs does).

> I will post later on my thoughts on how to control the system. Are
> people, in principle, willing to put magic incantations in their emails
> and commit messages? I'm not asking for a commitment to my tool here,
> I'm just exploring what the bounds of people's, and committer's in
> particular, willingness to adjust their workflow or message texts a bit
> to make it easier to automate bug tracking. Even if people don't
> want to use what I've done, the same problems arise for any system.

'magic incantations' are probably alright, up to a point- they generally
shouldn't be required for messages to be added to the bug.

The biggest issue that I see with building a new system like this rather
than using something which already exists is that dealing with email is
no trivial task, as we've seen with our archives system and with the
various other bug trackers that have been discussed (many of which don't
deal very well with email).

Thanks!

Stephen