Re: Unit testing

On Tue, 2004-10-12 at 05:08, Greg Stark wrote:
> But it seems to me that most of the really hard bugs to find involve subtle
> interactions between functions and the state of the database.
>
> You wouldn't be able to find errors in the semantics of xids for example, or
> in the WAL logic that didn't cover some corner case. Or race conditions
> between backends...

Going into this, these were precisely the kinds of bugs that Gavin and I
wanted to be able to find via some kind of automated QA. I agree that
unit tests aren't ideal for finding these kinds of bugs (although I
don't think they are useless), but what better technique is there?
Regression tests are certainly ineffective at best. Static analysis is
best for finding superficial bugs or enforcing invariants that are easy
to verify at compile-time, so even if there were good open source static
analysis tools I don't think it would be that helpful.

Model checking has some promise[1], but (a) it requires a substantial
amount of work to model check a program, even if we use a tool that will
automatically extract the model for us (e.g. CMC) (b) I'm not aware of a
good open source model checking tool (c) I'm skeptical that model
checking in general is mature enough that it is useful outside academia.

-Neil

[1] e.g. http://www.stanford.edu/~engler/osdi04-fisc.pdf