testing concurrency (was Re: subtransaction assert failure)

On Fri, 2004-09-17 at 01:43, Tom Lane wrote:
> PS: this points up once again that the regression tests do not do very
> well at testing concurrent behavior. We need to think about some sort
> of parallel-test harness that would let us write tests that have better
> odds of catching bugs like this.

I agree -- something like this would be very useful. I'm a little
stumped about how to actually write such a thing, though...

One incremental improvement to the existing RT system would be to make
each test self-contained (i.e. as far as possible it would assume
nothing about DBMS state before it ran, and not leave the DBMS state
modified when it finished running). If writing completely self-contained
tests isn't possible, we could specify a dependency graph for the tests.
Then we could have the regression test system generate a randomized
grouping of tests to be executed. In order to have reproducible results,
we could associate each randomized grouping with a numeric key (there
should be some way to produce a 1-to-1 mapping). So someone reporting a
regression test problem would say: "I see the following regression test
failures, the test key is xyz", and a developer could run the regression
tests specifying that key and have the RT system run the same
combination of tests in the same order that produced the problem the
user saw. This would be a complement to the existing method of running
the RTs.

I still think we need a separate way to test parallel behavior though,
in addition to the above idea. Can anyone think of a good way to do
this?

-Neil