Re: How can we make beta testing better?

On 04/17/14 15:16, Merlin Moncure wrote:
> On Tue, Apr 15, 2014 at 4:47 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>> Hackers,
>>
>> I think 9.3 has given us evidence that our users aren't giving new
>> versions of PostgreSQL substantial beta testing, or if they are, they
>> aren't sharing the results with us.
>>
>> How can we make beta testing better and more effective? How can we get
>> more users to actually throw serious workloads at new versions and share
>> the results?
>>
>> I've tried a couple of things over the last two years and they haven't
>> worked all that well. Since we're about to go into another beta testing
>> period, we need something new. Ideas?
>
> I've seen lots of bugs reported and fixed in the beta period over the
> years. My take is that it's basically unrealistic to expect volunteer
> beta testers to replace bone fide regression testing.
>
> I think it's a pretty fair statement that we've had some QC issues in
> the general area of replication technologies. What this is indicating
> to me is that replication needs substantially more coverage in 'make
> check'. Since I'm wishing for things, it would be nice to see an
> expansion of the buildfarm so that we could [optionally] run various
> performance tests as well as various replication scenarios. Then we
> could go back to users and say, please donate 'repeatable tests and
> machines to run them on' and reap the long term value.
>
> Not at all making light out of any of this...it's a huge project.

The problem with testing replication is that it doesn't fit well into
our standard regression testing. There are way too many moving parts as
well as dependencies on the underlying OS and network topology.

You will discover a ton of race conditions once you actually move from
testing with multiple postmasters (so you can kill one) on the same box
to using multiple virtual machines and instead of completely severing a
network connection using some packet shaping/filtering to introduce
packet loss, limited bandwidth, async routing and so on. At least that
is my experience from throwing that sort of sh*t at Slony at full speed.

Not trying to discourage anyone from trying. Just saying that it doesn't
fit into our existing regression test framework.

Jan

--
Jan Wieck
Senior Software Engineer
http://slony.info