Re: Unit testing

On Tue, 12 Oct 2004, Neil Conway wrote:

> On Tue, 2004-10-12 at 05:08, Greg Stark wrote:
> > But it seems to me that most of the really hard bugs to find involve subtle
> > interactions between functions and the state of the database.
> >
> > You wouldn't be able to find errors in the semantics of xids for example, or
> > in the WAL logic that didn't cover some corner case. Or race conditions
> > between backends...
>
> Going into this, these were precisely the kinds of bugs that Gavin and I
> wanted to be able to find via some kind of automated QA. I agree that
> unit tests aren't ideal for finding these kinds of bugs (although I
> don't think they are useless), but what better technique is there?
> Regression tests are certainly ineffective at best. Static analysis is
> best for finding superficial bugs or enforcing invariants that are easy
> to verify at compile-time, so even if there were good open source static
> analysis tools I don't think it would be that helpful.

I agree. The fact is, doing nothing new means that we will not find bugs
which the regression test system is unsuited to finding. This makes the
question(s), will adding unit tests: i) find enough current bugs
(presumably, quite a small number); and, ii) assist with development in
the future to the extent that it is worth spending the time working on the
tests themselves. I'm unsure of the worth of arguing this in abstract.
I'll start looking at the amount of time involved in testing functions in
different parts of the source and then report in the coming weeks.

> Model checking has some promise[1], but (a) it requires a substantial
> amount of work to model check a program, even if we use a tool that will
> automatically extract the model for us (e.g. CMC) (b) I'm not aware of a
> good open source model checking tool (c) I'm skeptical that model
> checking in general is mature enough that it is useful outside academia.

Given infinite resources, I'd say it would be worthwhile building a model
checking system analogous to Engler's for Postgres. That is, see if each
data modification operation (ie, additional to WAL) via 'all' possible
paths was recoverable and that the recovery placed the system in the state
expected. Based on the information in Engler's paper, I'd say this would
take a few man months for a very clued up developer. Anyone out there
looking for a thesis project? :-)

Gavin