Re: New regression test time

-----Original Message-----
From: pgsql-hackers-owner(at)postgresql(dot)org [mailto:pgsql-hackers-owner(at)postgresql(dot)org] On Behalf Of Josh Berkus
Sent: Saturday, June 29, 2013 3:00 PM
To: Andrew Dunstan
Cc: Alvaro Herrera; pgsql-hackers(at)postgresql(dot)org; Robins Tharakan
Subject: Re: [HACKERS] New regression test time

On 06/29/2013 02:14 PM, Andrew Dunstan wrote:
> AIUI: They do test feature use and errors that have cropped up in the
> past that we need to beware of. They don't test every bug we've ever
> had, nor do they exercise every piece of code.

If we don't have a test for it, then we can break it in the future and not know we've broken it until .0 is released. Is that really a direction we're happy going in?

>
> Maybe there is a good case for these last two in a different set of tests.

If we had a different set of tests, that would be a valid argument. But we don't, so it's not. And nobody has offered to write a feature to split our tests either.

I have to say, I'm really surprised at the level of resistance people on this list are showing to the idea of increasing test coverage. I thought
that Postgres was all about reliability? For a project as mature as we
are, our test coverage is abysmal, and I think I'm starting to see why.
>>
An ounce of prevention is worth a pound of cure.
The cost of a bug rises exponentially, starting at requirements gathering and ending at the customer.
Where I work, we have two computer rooms full of machines that run tests around the clock, 24x365.
Even so, a full regression takes well over a week because we perform hundreds of thousands of tests.
All choices of this kind are trade-offs. But in such situations, my motto is:
"Do whatever will make the customer prosper the most."
IMO-YMMV
<<