Re: Two weeks to feature freeze

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> Sent: Friday, June 20, 2003 8:58 PM
> To: Jason Earl
> Cc: Dann Corbit; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
>
>
> Jason Earl <jason(dot)earl(at)simplot(dot)com> writes:
> > Hmm... I must have missed the huge corporation paying for in house
> > testing of PostgreSQL. In the Free Software world the
> "beta team" is
> > all of those people that need the new features so badly
> that they are
> > willing to risk their own data and hardware testing it.
>
> I don't have a lot of faith in huge automated test efforts.
> They're great at ensuring you don't make the same mistakes
> you made once before, but in my experience the nastiest bugs
> are the ones you haven't seen before and would never in a
> million years have dreamed to test for.

This is true if and only if the test design is poor.

> Thus, the best test
> team is a bunch of people doing unplanned things with the
> software, on a wide variety of platforms...

That is the worst possible test plan. It totally lacks organization and
there is no hint to define when the feature set has been covered. Ad
hoc testing is a useful addition, but it cannot replace all the standard
tests that have been used by the industry for decades.

If you run literally hundreds of tests designed to ensure that your
product conforms to ANSI/ISO standards then the bugs that are missed
will be few and far between. Unless you are bad at designing tests.

Designing tests is busywork. Desiging tests is boring. Nobody wants to
design tests, let alone interpret the results and define correct
baselines. But testing is very, very important.