Re: How can we make beta testing better?

On Tue, Apr 15, 2014 at 2:47 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:

> Hackers,
>
> I think 9.3 has given us evidence that our users aren't giving new
> versions of PostgreSQL substantial beta testing, or if they are, they
> aren't sharing the results with us.
>

A lot of the bugs that turned up are not the kind I would expect to have
been found in most beta testing done by non-hacking users. Weren't they
mostly around rare race conditions, crash recovery, and freezing?

>
> How can we make beta testing better and more effective? How can we get
> more users to actually throw serious workloads at new versions and share
> the results?
>

If we are interested in positive results as well as negative, we should
change https://wiki.postgresql.org/wiki/HowToBetaTest

"pgsql-hackers: bugs, questions, and successful test reports are welcome
here if you are already subscribed to pgsql-hackers. Note that
pgsql-hackers is a high-traffic mailing list with a lot of development
discussion."

So successful reports are welcome, provided that you are willing to
subscribe to a list that generates tons of noise you won't understand.
That doesn't sound all that welcoming. (I already am subscribed, but I
still usually don't report successful tests, because "yeah, I did a bunch
of stuff, and nothing failed in an obvious way" just doesn't sound very
useful, and it is hard to get motivated to write up an exhaustive
description of a test that doesn't prove anything anyway--maybe if I did
for a few more hours, it would have found a problem.)

If we want to know how much beta testing is really going on, perhaps we
could do a survey asking people whether they did any beta testing, and if
so whether they reported the results. Otherwise it would be hard to
distinguish "We aren't doing enough testing" from "We do lots of testing,
but it isn't strenuous enough to find the problems, or is testing the wrong
aspects of the system".

Cheers,

Jeff