Re: New regression test time

On Sat, Jun 29, 2013 at 02:59:35PM -0700, Josh Berkus wrote:
> On 06/29/2013 02:14 PM, Andrew Dunstan wrote:
> > AIUI: They do test feature use and errors that have cropped up in the
> > past that we need to beware of. They don't test every bug we've ever
> > had, nor do they exercise every piece of code.
>
> If we don't have a test for it, then we can break it in the future and
> not know we've broken it until .0 is released. Is that really a
> direction we're happy going in?
>
> > Maybe there is a good case for these last two in a different set of tests.
>
> If we had a different set of tests, that would be a valid argument. But
> we don't, so it's not. And nobody has offered to write a feature to
> split our tests either.

With utmost respect, this just isn't true. There is a "make coverage"
target that probably doesn't get enough exercise, but it's just the
kind of infrastructure you're describing.

> I have to say, I'm really surprised at the level of resistance
> people on this list are showing to the idea of increasing test
> coverage. I thought that Postgres was all about reliability? For a
> project as mature as we are, our test coverage is abysmal, and I
> think I'm starting to see why.

Burdening hackers with extra time in ordinary compile cycles is the
wrong direction. If anything, we should probably look at what tests
only routinely get run by our CI system--currently the buildfarm--and
which ones developers could reasonably be expected to wait until
post-push to run in day-to-day development.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate