Re: TAP tests take a long time

On 04/11/2017 08:12 PM, Craig Ringer wrote:
>
>
> On 12 Apr. 2017 03:14, "Andrew Dunstan"
> <andrew(dot)dunstan(at)2ndquadrant(dot)com
> <mailto:andrew(dot)dunstan(at)2ndquadrant(dot)com>> wrote:
>
>
>
> On 04/11/2017 12:08 PM, Tom Lane wrote:
> > Robert Haas <robertmhaas(at)gmail(dot)com
> <mailto:robertmhaas(at)gmail(dot)com>> writes:
> >> On Tue, Apr 11, 2017 at 11:32 AM, Andrew Dunstan
> >> <andrew(dot)dunstan(at)2ndquadrant(dot)com
> <mailto:andrew(dot)dunstan(at)2ndquadrant(dot)com>> wrote:
> >>> This buildfarm run as you can see takes 33m32s, and the Tap
> tests take a
> >>> combined 19m52s of that time.
> >> I don't think it's quite fair to complain about the TAP tests
> taking a
> >> long time to run as a general matter. Many people here have long
> >> wanted a separate test-suite for long running tests that can be
> run to
> >> really check everything possible, and apparently, TAP tests are it.
> >> What I think would be more useful is to break down where the
> time is
> >> getting spent. It may be that some of those tests are not adding
> >> value proportionate to their runtime.
>
>
>
> Well, you can get a lot of information on timings in crake's latest
> builds for example, or nightjar's.
>
> Here's an interesting fact: almost all the time taken up in the
> scripts
> test set seems to be consumed in running initdb over and over.
>
>
>
> Is it worth adding an init(cache => 1) option to PostgresNode, where
> we stash an initdb'd db in tmp_check/ and just do a simple fs copy of
> it ?
>
> Even default to caching and allow an option to disable the cached copy.
>
> We're going to need to initdb a lot in the TAP tests. We often need a
> clean state. Tests also get harder to debug the more you pack into a
> single test run and more difficult to run individually to test some
> specific failure. So IMO the best thing is to try to make that repeat
> initdb as fast as possible.
>
> It reduces our coverage of initdb only incredibly slightly - all that
> repeat runs will do is help find very uncommon intermittent failures.
> And we rerun the buildfarm all the time so it's not like there's a
> shortage of initdb runs anyway.
>
> We should also have a debug --no-fsync option for initdb, or maybe
> allow it to take -o options to pass to child postgres so we can pass
> fsync=off .

Absolutely worth it I should say.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services