Re: Feature test issues

The team here at Pivotal is doing TDD already. Running so many
configurations sounds like it belongs in a CI not in a TDD cycle.

I want to make sure we are making the most effective technical investment
that we can. My thought process is generally:
* Are we feeling pain from this right now? (is it slowing down
development, leading to confusion, bugs, etc?)*

*If not, are we painting ourselves into a corner? (Will it be an order of
magnitude harder to make this investment later?)*

*How does this pain compare to other sources of technical pain?*

Once we've decided to invest in easing a pain, I find it valuable to step
back and try to see if there are other options that might also address it.

In the case of running multiple configurations, I suspect you might get a
2 or 3x speedup of the entire suite by running tests in parallel.
The app startup time is pretty fast. It takes about two to three seconds.
At this point, that would shave about 6 seconds per suite run or 30 seconds
in your 5-server run vs 3-4 minutes that I expect would be saved through
parallelization.

Regarding the python versions, have we considered packaging the app bundle
with the latest version of python that we support? Then we could run tests
with that version and the latest version of python that we are working
towards supporting. That would save us the headache of trying to straddle
the 2.x/3.x language gap.

Tira

On Mon, Feb 27, 2017, 4:53 AM Dave Page <dpage(at)pgadmin(dot)org> wrote:

> On Mon, Feb 27, 2017 at 4:02 AM, Atira Odhner <aodhner(at)pivotal(dot)io> wrote:
> > Cool, we already have a task about proper teardown and a couple other
> things
> > in our backlog. we'll probably get to it in the next day or so. I'll
> take a
> > look at the other stuff.
>
> Thanks.
>
> > Also, regarding speed, even without the app startup time, end to end
> tests
> > are always going to be relatively slow. We definitely want to make sure
> that
> > the time it takes to run the tests does not grow to where it is a
> deterrent
> > to running them.
>
> Right - I expect them to be slower, but 1.5+ minutes per test (with 5
> DB servers - we're soon going to have 10+) is not going to work. I
> want to get us to the point where we're doing test driven development,
> with the aim of always having the tree in a releasable state.
>
> > There are a variety of things we can do to help address that as our suite
> > grows. For instance, we could consider parallelizing the tests, making
> > setup and teardown more efficient, combining related tests, or even
> > breaking the tests into suites and running only some of them locally by
> > default.
> > Since we only have a couple feature tests so far the speed hasn't really
> > felt like an issue for me yet, but I understand it may be different if
> you
> > are trying to run in a variety of configurations.
>
> I'm looking ahead to where we want to be. I don't want the test suite
> to become a source of technical debt.
>
> > Out of curiosity, what is the goal in supporting multiple python
> versions?
> > Are we working on moving to 3.x and just haven't gotten fully there yet?
>
> We need to support multiple versions of Python because that's what
> users have on their systems. For example, RHEL/CentOS 6 which are
> still in wide use ship with Python 2.6.
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>