Re: Test code is worth the space

On 8/16/15 8:48 AM, Greg Stark wrote:
> On Sun, Aug 16, 2015 at 7:33 AM, Noah Misch <noah(at)leadboat(dot)com> wrote:
>> When I've just spent awhile implementing a behavior change, the test diffs are
>> a comforting sight. They confirm that the test suite exercises the topic I
>> just changed. Furthermore, most tests today do not qualify under this
>> stringent metric you suggest. The nature of pg_regress opposes it.
>
> It's not a comforting sight for me. It means I need to change the test
> results and then of course the tests will pass. So they didn't really
> test anything and I don't really know whether the changes broke
> anything. Any test I had to adjust for my change was effectively
> worthless.
>
> This is precisely my point about pg_regress and why it's the reason
> there's pressure not to have extensive tests. It's useful to have some
> end-to-end black box tests like this but it's self-limiting. You can't
> test many details without tying the tests to specific behaviour.
>
> I have worked with insanely extensive testing suites that didn't have
> this problem. But they were unit tests for code that was structured
> around testability. When the API is designed to be testable and you
> have good infrastructure for mocking up the environment and comparing
> function results in a much narrower way than just looking at text
> output you can test the correctness of each function without tying the
> test to the specific behaviour.
>
> *That* gives a real comfort. When you change a function to behave
> entirely differently and can still run all the tests and see that all
> the code that depends on it still operate correctly then you know you
> haven't broken anything. In that case it actually *speeds up*
> development rather than slowing it down. It lets newbie developers
> hack on code where they don't necessarily know all the downstream
> dependent code and not be paralyzed by fear that they'll break
> something.

As someone who oversaw a database test suite with almost 500 test files
(each testing multiple cases), I completely agree. Our early framework
was actually similar to pg_regress and that worked "OK" up to about 200
test files, but it got very unwieldy very quickly. We switched to pgTap
and it was far easier to work with.

I suspect any effort to significantly improve Postgres test coverage is
doomed until there's an alternative to pg_regress.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Data in Trouble? Get it in Treble! http://BlueTreble.com