Re: testing concurrency (was Re: subtransaction assert failure)

Neil Conway <neilc(at)samurai(dot)com> writes:
> On Fri, 2004-09-17 at 01:43, Tom Lane wrote:
>> PS: this points up once again that the regression tests do not do very
>> well at testing concurrent behavior. We need to think about some sort
>> of parallel-test harness that would let us write tests that have better
>> odds of catching bugs like this.

> One incremental improvement to the existing RT system would be to make
> each test self-contained (i.e. as far as possible it would assume
> nothing about DBMS state before it ran, and not leave the DBMS state
> modified when it finished running). If writing completely self-contained
> tests isn't possible, we could specify a dependency graph for the tests.
> Then we could have the regression test system generate a randomized
> grouping of tests to be executed.

This is pretty much what we have now, and it's quite useless, because
the tests that are run concurrently are very careful to stay out of each
others' way. There are certainly no tests that deliberately exercise
concurrent-update scenarios, and AFAICS the scheme you are suggesting
would not do that either.

There are some speculations in the archives about writing a test engine
that could fire up multiple database connections and then issue specific
command sequences across the set of connections. It's clear that you
could test questions like does-this-lock-really-block-that-operation
that way. I'm not sure that this approach does anything much to catch
problems with small windows-of-opportunity though :-(

Mulling it over ... it seems the real issue is that the regression tests
are designed for reproducibility, whereas what we'd really like to probe
for this sort of bug is *non* reproducible behavior. Not sure how to
approach that, except that maybe we shouldn't think in terms of tests
that every Joe Bob and Harry can run and not need any intelligence to
interpret the results of ...

regards, tom lane