Re: Unit testing

Neil Conway <neilc(at)samurai(dot)com> writes:
> On Tue, 2004-10-12 at 05:08, Greg Stark wrote:
>> You wouldn't be able to find errors in the semantics of xids for example, or
>> in the WAL logic that didn't cover some corner case. Or race conditions
>> between backends...

> Going into this, these were precisely the kinds of bugs that Gavin and I
> wanted to be able to find via some kind of automated QA. I agree that
> unit tests aren't ideal for finding these kinds of bugs (although I
> don't think they are useless), but what better technique is there?
> Regression tests are certainly ineffective at best.

Ahem. Our *existing* regression tests are fairly ineffective, but
that's because neither the test cases nor the engine are designed to
cover concurrent behavior at all; if anything they go out of their way
to avoid stressing concurrent behavior, in order to get perfectly
constant results.

We've speculated in the past about building a test harness that could
step multiple backends through a concurrent script. Building something
like that, perhaps with some extra frammishes such as being able to
automatically vary the relative timing of operations, is the best
testing idea that I've heard about. Also you could extend it to force
crashes at varying points in the sequence and check for successful
recovery (which could catch WAL omissions such as Greg was worrying
about). You could probably build this on top of an existing tool like
"expect".

While you've not said much about exactly what you have in mind for your
unit-test scheme, I doubt it will be of any value at all if it doesn't
provide ways to test concurrent behavior. I'm also quite concerned
about the cost of building scaffolding that will allow individual
modules to be tested outside the context of a live backend; and more
than a bit dubious about the effectiveness of tests in a scaffolding
environment instead of a real one.

regards, tom lane