Re: Concurrency testing

On Wed, 7 Oct 2009, Alvaro Herrera wrote:

> Yeah, the API they implemented wasn't ideal, so there was some
> discussion that ended up with a specification everyone was happy with,
> but then nobody got around to implementing it.

I needed something like this and didn't implement those suggestions
because I thought the whole idea didn't scale up enough. That's close to
the right API to allow more complicated regression tests in psql itself,
but I doubted that would hit real complexity level needed to find the good
concurrent bugs.

The stuff I've been building lately takes "how many sessions at once?" as
an input, and the useful range on that (particularly as we wander further
toward multi-core land) is much higher than you'd want to manage one at a
time manually. I'd rather see a "session simulator" program that picks
among several types of behaviors and executes them, and then you can throw
as many of those as you want into the mix. You can do something like that
with pgbench right now if you pass it multiple files, the tricky part is
figuring out what regression output you should expect.

We had a quick session on more complicated PG testing at PGCon, and it
seems many of us who looked at this from multiple perspectives have
settled into the idea that the current regression tests should stay as
they are, and the more complicated ones are going to need to be in Perl to
keep the code complexity managable but still allow more complicated tests.
Another example of a complicted test is something that tests
pg_dump/pg_restore, which is also hard to shoehorn into pg_regress. I
personally would rather see progress made in that area than trying to hack
increasingly difficult tests into the existing regression framework,
particular given how changes there impact existing committer's daily
workflow. The complexity level of test you need to find the juicy
concurrent issues is not one you're going to want to run every time you do
"make check".

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD