Re: How can we make beta testing better?

On Tue, Apr 15, 2014 at 4:47 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> Hackers,
>
> I think 9.3 has given us evidence that our users aren't giving new
> versions of PostgreSQL substantial beta testing, or if they are, they
> aren't sharing the results with us.
>
> How can we make beta testing better and more effective? How can we get
> more users to actually throw serious workloads at new versions and share
> the results?
>
> I've tried a couple of things over the last two years and they haven't
> worked all that well. Since we're about to go into another beta testing
> period, we need something new. Ideas?

I've seen lots of bugs reported and fixed in the beta period over the
years. My take is that it's basically unrealistic to expect volunteer
beta testers to replace bone fide regression testing.

I think it's a pretty fair statement that we've had some QC issues in
the general area of replication technologies. What this is indicating
to me is that replication needs substantially more coverage in 'make
check'. Since I'm wishing for things, it would be nice to see an
expansion of the buildfarm so that we could [optionally] run various
performance tests as well as various replication scenarios. Then we
could go back to users and say, please donate 'repeatable tests and
machines to run them on' and reap the long term value.

Not at all making light out of any of this...it's a huge project.

merlin