Re: Need a builtin way to run all tests faster manner

On 03/13/2017 12:35 AM, Tom Lane wrote:
> Andres Freund <andres(at)anarazel(dot)de> writes:
>> On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
>>> This looks generally sane to me, although I'm not very happy about folding
>>> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
>>> seems weird and unlike the way it's done for the regular regression test
>>> case.
>> Yea, not super happy about that either - alternatively we could fold it
>> into pg_regress.
> Yeah, teaching pg_regress to auto-create the --temp-instance directory
> seems perfectly sane from here.

w.r.t. $subject, I thought it might be useful to get some recent stats
from the buildfarm. Results are below. The bin checks dwarf everything
else. Upgrade checks and isolation check are other items of significant
cost. Upgrade checks could be significantly shortened if we could avoid
rerunning the regression tests.

cheers

andrew

pgbfprod=> select s.branch, s.log_stage, count(*), avg(extract(epoch
from stage_duration)::numeric(15,2))::numeric(15,1),
stddev(extract(epoch from stage_duration)::numeric(15,2))::numeric(15,1)
from public.build_status_log s where sysname <> 'crake' and snapshot >
now() - interval '10 days' and log_stage !~ 'start|stop' group by
s.branch, s.log_stage having count(*) > 20 and avg(extract(epoch from
stage_duration)::numeric(15,2)) > 20 order by log_stage, branch <>
'HEAD', branch desc
;
branch | log_stage | count
| avg | stddev
---------------+-----------------------------------------------+-------+--------+--------
HEAD | bin-check.log | 388 |
1739.0 | 1758.5
REL9_6_STABLE | bin-check.log | 91 |
1430.5 | 1287.9
REL9_5_STABLE | bin-check.log | 87 |
1140.0 | 994.1
REL9_4_STABLE | bin-check.log | 72
| 751.0 | 666.8
HEAD | check.log | 2305
| 263.1 | 1197.0
REL9_6_STABLE | check.log | 610
| 294.7 | 1369.6
REL9_5_STABLE | check.log | 627
| 170.1 | 819.6
REL9_4_STABLE | check.log | 512
| 140.4 | 535.3
REL9_3_STABLE | check.log | 449
| 112.0 | 446.0
REL9_2_STABLE | check.log | 406
| 109.2 | 380.9
HEAD | check-pg_upgrade.log | 1785
| 319.4 | 1310.5
REL9_6_STABLE | check-pg_upgrade.log | 482
| 571.3 | 2811.0
REL9_5_STABLE | check-pg_upgrade.log | 484
| 350.5 | 2160.3
REL9_4_STABLE | check-pg_upgrade.log | 385
| 240.8 | 1278.9
REL9_3_STABLE | check-pg_upgrade.log | 353
| 214.0 | 1188.3
REL9_2_STABLE | check-pg_upgrade.log | 314
| 195.6 | 1016.6
HEAD | config.log | 2216
| 84.5 | 101.5
REL9_6_STABLE | config.log | 576
| 90.0 | 90.8
REL9_5_STABLE | config.log | 584
| 114.0 | 358.8
REL9_4_STABLE | config.log | 495
| 84.5 | 85.3
REL9_3_STABLE | config.log | 431
| 97.9 | 100.7
REL9_2_STABLE | config.log | 391
| 93.1 | 94.6
HEAD | contrib-install-check-C.log | 2250
| 122.9 | 474.5
REL9_6_STABLE | contrib-install-check-C.log | 606
| 124.6 | 410.3
REL9_5_STABLE | contrib-install-check-C.log | 622
| 84.7 | 348.8
REL9_4_STABLE | contrib-install-check-C.log | 508
| 105.9 | 434.1
REL9_3_STABLE | contrib-install-check-C.log | 445
| 61.8 | 273.4
REL9_2_STABLE | contrib-install-check-C.log | 403
| 54.3 | 205.7
HEAD | contrib-install-check-cs_CZ.UTF-8.log | 184
| 25.7 | 11.8
REL9_6_STABLE | contrib-install-check-cs_CZ.UTF-8.log | 46
| 25.7 | 14.0
REL9_5_STABLE | contrib-install-check-cs_CZ.UTF-8.log | 42
| 23.8 | 16.6
REL9_4_STABLE | contrib-install-check-cs_CZ.UTF-8.log | 36
| 22.9 | 12.4
HEAD | contrib-install-check-en_US.8859-15.log | 37
| 173.9 | 32.3
HEAD | contrib-install-check-en_US.ISO8859-1.log | 33
| 244.7 | 35.7
HEAD | contrib-install-check-en_US.log | 171
| 65.9 | 101.6
REL9_6_STABLE | contrib-install-check-en_US.log | 42
| 61.7 | 89.3
REL9_5_STABLE | contrib-install-check-en_US.log | 37
| 54.3 | 79.3
REL9_4_STABLE | contrib-install-check-en_US.log | 33
| 53.7 | 71.5
REL9_3_STABLE | contrib-install-check-en_US.log | 32
| 57.8 | 86.6
REL9_2_STABLE | contrib-install-check-en_US.log | 29
| 46.9 | 63.8
HEAD | contrib-install-check-en_US.utf8.log | 142
| 29.2 | 31.4
REL9_6_STABLE | contrib-install-check-en_US.utf8.log | 30
| 40.2 | 43.7
REL9_5_STABLE | contrib-install-check-en_US.utf8.log | 29
| 41.8 | 51.6
REL9_4_STABLE | contrib-install-check-en_US.utf8.log | 25
| 56.6 | 75.8
REL9_3_STABLE | contrib-install-check-en_US.utf8.log | 24
| 35.7 | 40.4
REL9_2_STABLE | contrib-install-check-en_US.utf8.log | 21
| 31.3 | 36.1
HEAD | contrib-install-check-en_US.UTF-8.log | 188
| 79.7 | 73.8
REL9_6_STABLE | contrib-install-check-en_US.UTF-8.log | 36
| 78.2 | 68.6
REL9_5_STABLE | contrib-install-check-en_US.UTF-8.log | 34
| 64.9 | 53.7
REL9_4_STABLE | contrib-install-check-en_US.UTF-8.log | 30
| 68.4 | 56.2
REL9_3_STABLE | contrib-install-check-en_US.UTF-8.log | 28
| 54.6 | 46.1
REL9_2_STABLE | contrib-install-check-en_US.UTF-8.log | 25
| 48.3 | 39.8
HEAD | contrib-install-check-POSIX.log | 33
| 227.7 | 42.6
HEAD | ecpg-check.log | 2229
| 88.5 | 212.4
REL9_6_STABLE | ecpg-check.log | 586
| 97.5 | 139.6
REL9_5_STABLE | ecpg-check.log | 591
| 92.5 | 183.5
REL9_4_STABLE | ecpg-check.log | 491
| 82.1 | 117.1
REL9_3_STABLE | ecpg-check.log | 428
| 77.3 | 102.8
REL9_2_STABLE | ecpg-check.log | 388
| 80.7 | 98.3
REL9_6_STABLE | initdb-C.log | 606
| 21.2 | 73.1
HEAD | initdb-en_US.8859-15.log | 37
| 241.8 | 24.8
HEAD | initdb-en_US.ISO8859-1.log | 33
| 251.6 | 39.1
HEAD | initdb-en_US.log | 171
| 54.3 | 103.8
REL9_6_STABLE | initdb-en_US.log | 42
| 54.9 | 99.7
REL9_5_STABLE | initdb-en_US.log | 37
| 61.2 | 112.5
REL9_4_STABLE | initdb-en_US.log | 33
| 47.5 | 78.5
REL9_3_STABLE | initdb-en_US.log | 32
| 46.0 | 74.6
REL9_2_STABLE | initdb-en_US.log | 29
| 41.1 | 62.9
HEAD | initdb-POSIX.log | 33
| 251.1 | 31.6
HEAD | install-check-C.log | 2241
| 229.5 | 1227.9
REL9_6_STABLE | install-check-C.log | 606
| 299.0 | 1601.9
REL9_5_STABLE | install-check-C.log | 622
| 173.7 | 1147.2
REL9_4_STABLE | install-check-C.log | 508
| 139.3 | 796.4
REL9_3_STABLE | install-check-C.log | 446
| 104.7 | 713.3
REL9_2_STABLE | install-check-C.log | 403
| 91.3 | 574.6
HEAD | install-check-cs_CZ.ISO-8859-2.log | 137
| 37.5 | 1.5
REL9_6_STABLE | install-check-cs_CZ.ISO-8859-2.log | 33
| 30.8 | 1.5
REL9_5_STABLE | install-check-cs_CZ.ISO-8859-2.log | 29
| 31.0 | 1.7
REL9_4_STABLE | install-check-cs_CZ.ISO-8859-2.log | 25
| 26.2 | 0.4
REL9_3_STABLE | install-check-cs_CZ.ISO-8859-2.log | 24
| 24.9 | 1.2
REL9_2_STABLE | install-check-cs_CZ.ISO-8859-2.log | 21
| 22.6 | 0.7
HEAD | install-check-cs_CZ.UTF-8.log | 184
| 43.0 | 9.7
REL9_6_STABLE | install-check-cs_CZ.UTF-8.log | 46
| 39.1 | 15.1
REL9_5_STABLE | install-check-cs_CZ.UTF-8.log | 42
| 39.4 | 14.6
REL9_4_STABLE | install-check-cs_CZ.UTF-8.log | 36
| 32.3 | 9.5
REL9_3_STABLE | install-check-cs_CZ.UTF-8.log | 34
| 29.7 | 8.2
REL9_2_STABLE | install-check-cs_CZ.UTF-8.log | 30
| 27.6 | 9.2
HEAD | install-check-cs_CZ.WIN-1250.log | 137
| 37.3 | 1.2
REL9_6_STABLE | install-check-cs_CZ.WIN-1250.log | 33
| 30.5 | 1.3
REL9_5_STABLE | install-check-cs_CZ.WIN-1250.log | 29
| 31.0 | 1.7
REL9_4_STABLE | install-check-cs_CZ.WIN-1250.log | 25
| 26.3 | 0.7
REL9_3_STABLE | install-check-cs_CZ.WIN-1250.log | 24
| 24.9 | 1.4
REL9_2_STABLE | install-check-cs_CZ.WIN-1250.log | 21
| 22.3 | 0.6
HEAD | install-check-de_DE.utf8.log | 129
| 35.4 | 9.4
REL9_6_STABLE | install-check-de_DE.utf8.log | 22
| 25.8 | 7.8
REL9_5_STABLE | install-check-de_DE.utf8.log | 21
| 27.4 | 8.0
HEAD | install-check-en_US.8859-15.log | 37
| 136.9 | 21.2
HEAD | install-check-en_US.ISO8859-1.log | 33
| 319.0 | 29.4
HEAD | install-check-en_US.log | 171
| 65.7 | 58.7
REL9_6_STABLE | install-check-en_US.log | 42
| 62.7 | 69.4
REL9_5_STABLE | install-check-en_US.log | 37
| 59.4 | 56.2
REL9_4_STABLE | install-check-en_US.log | 33
| 50.6 | 44.7
REL9_3_STABLE | install-check-en_US.log | 32
| 50.2 | 46.8
REL9_2_STABLE | install-check-en_US.log | 29
| 45.1 | 39.2
HEAD | install-check-en_US.utf8.log | 142
| 47.6 | 40.3
REL9_6_STABLE | install-check-en_US.utf8.log | 30
| 58.1 | 54.4
REL9_5_STABLE | install-check-en_US.utf8.log | 29
| 61.7 | 57.4
REL9_4_STABLE | install-check-en_US.utf8.log | 25
| 57.2 | 59.9
REL9_3_STABLE | install-check-en_US.utf8.log | 24
| 51.3 | 47.7
REL9_2_STABLE | install-check-en_US.utf8.log | 21
| 45.2 | 41.9
HEAD | install-check-en_US.UTF-8.log | 188
| 153.5 | 162.7
REL9_6_STABLE | install-check-en_US.UTF-8.log | 36
| 140.8 | 140.9
REL9_5_STABLE | install-check-en_US.UTF-8.log | 34
| 145.7 | 144.3
REL9_4_STABLE | install-check-en_US.UTF-8.log | 30
| 118.9 | 113.1
REL9_3_STABLE | install-check-en_US.UTF-8.log | 28
| 107.0 | 101.8
REL9_2_STABLE | install-check-en_US.UTF-8.log | 25
| 93.6 | 87.8
HEAD | install-check-POSIX.log | 33
| 189.5 | 76.5
HEAD | install-check-sk_SK.ISO-8859-2.log | 137
| 37.4 | 1.2
REL9_6_STABLE | install-check-sk_SK.ISO-8859-2.log | 33
| 30.5 | 1.3
REL9_5_STABLE | install-check-sk_SK.ISO-8859-2.log | 29
| 30.9 | 1.5
REL9_4_STABLE | install-check-sk_SK.ISO-8859-2.log | 25
| 26.2 | 0.4
REL9_3_STABLE | install-check-sk_SK.ISO-8859-2.log | 24
| 24.8 | 1.2
REL9_2_STABLE | install-check-sk_SK.ISO-8859-2.log | 21
| 22.7 | 0.6
HEAD | install-check-sk_SK.UTF-8.log | 137
| 37.6 | 1.1
REL9_6_STABLE | install-check-sk_SK.UTF-8.log | 33
| 31.0 | 1.8
REL9_5_STABLE | install-check-sk_SK.UTF-8.log | 29
| 31.3 | 1.8
REL9_4_STABLE | install-check-sk_SK.UTF-8.log | 25
| 26.2 | 0.4
REL9_3_STABLE | install-check-sk_SK.UTF-8.log | 24
| 25.0 | 1.2
REL9_2_STABLE | install-check-sk_SK.UTF-8.log | 21
| 22.7 | 1.2
HEAD | install-check-sk_SK.WIN-1250.log | 137
| 37.4 | 1.2
REL9_6_STABLE | install-check-sk_SK.WIN-1250.log | 33
| 30.4 | 0.7
REL9_5_STABLE | install-check-sk_SK.WIN-1250.log | 29
| 31.0 | 1.7
REL9_4_STABLE | install-check-sk_SK.WIN-1250.log | 25
| 26.2 | 0.4
REL9_3_STABLE | install-check-sk_SK.WIN-1250.log | 24
| 25.0 | 1.2
REL9_2_STABLE | install-check-sk_SK.WIN-1250.log | 21
| 22.8 | 1.1
HEAD | isolation-check.log | 2250
| 175.6 | 457.5
REL9_6_STABLE | isolation-check.log | 606
| 201.0 | 527.7
REL9_5_STABLE | isolation-check.log | 622
| 172.3 | 846.7
REL9_4_STABLE | isolation-check.log | 508
| 146.0 | 602.3
REL9_3_STABLE | isolation-check.log | 445
| 114.7 | 504.4
REL9_2_STABLE | isolation-check.log | 403
| 57.2 | 300.6
HEAD | make-contrib.log | 2176
| 43.7 | 67.6
REL9_6_STABLE | make-contrib.log | 572
| 46.2 | 50.3
REL9_5_STABLE | make-contrib.log | 579
| 44.9 | 54.2
REL9_4_STABLE | make-contrib.log | 491
| 50.0 | 52.3
REL9_3_STABLE | make-contrib.log | 428
| 48.6 | 51.4
REL9_2_STABLE | make-contrib.log | 388
| 42.2 | 42.8
HEAD | make-dist.log | 36
| 56.7 | 0.7
HEAD | make-doc.log | 36
| 150.8 | 1.5
REL9_2_STABLE | make-install.log | 403
| 22.0 | 29.8
HEAD | make.log | 2328
| 363.1 | 581.0
REL9_6_STABLE | make.log | 610
| 369.6 | 400.9
REL9_5_STABLE | make.log | 627
| 392.8 | 556.8
REL9_4_STABLE | make.log | 512
| 335.3 | 329.4
REL9_3_STABLE | make.log | 449
| 320.0 | 311.0
REL9_2_STABLE | make.log | 406
| 299.8 | 276.3
HEAD | pl-install-check-en_US.8859-15.log | 37
| 32.9 | 3.3
HEAD | pl-install-check-en_US.ISO8859-1.log | 33
| 34.3 | 5.1
HEAD | pl-install-check-POSIX.log | 33
| 33.1 | 8.3
REL9_2_STABLE | SCM-checkout.log | 406
| 33.2 | 87.8
HEAD | sepgsql-test.log | 63
| 20.7 | 6.8
HEAD | test-decoding-check.log | 1905
| 144.9 | 770.6
REL9_6_STABLE | test-decoding-check.log | 493
| 226.3 | 1244.0
REL9_5_STABLE | test-decoding-check.log | 495
| 147.1 | 1080.2
REL9_4_STABLE | test-decoding-check.log | 419
| 168.1 | 1117.6
HEAD | testmodules-install-check-C.log | 2233
| 22.6 | 42.3
REL9_6_STABLE | testmodules-install-check-C.log | 602
| 25.6 | 35.8
HEAD | testmodules-install-check-en_US.8859-15.log | 37
| 57.8 | 5.2
HEAD | testmodules-install-check-en_US.ISO8859-1.log | 33
| 58.9 | 10.6
HEAD | testmodules-install-check-en_US.log | 171
| 20.2 | 28.7
REL9_6_STABLE | testmodules-install-check-en_US.utf8.log | 30
| 30.4 | 46.9
HEAD | testmodules-install-check-en_US.UTF-8.log | 188
| 22.8 | 19.4
REL9_6_STABLE | testmodules-install-check-en_US.UTF-8.log | 36
| 23.6 | 19.4
REL9_5_STABLE | testmodules-install-check-en_US.UTF-8.log | 34
| 20.4 | 16.4
HEAD | testmodules-install-check-POSIX.log | 33
| 72.1 | 14.7
HEAD | typedefs.log | 96
| 74.8 | 99.1
REL9_6_STABLE | typedefs.log | 28
| 106.3 | 119.7
REL9_5_STABLE | typedefs.log | 28
| 100.0 | 111.7
REL9_4_STABLE | typedefs.log | 24
| 88.1 | 100.7
REL9_3_STABLE | typedefs.log | 23
| 85.6 | 95.1
(180 rows)

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services