Re: Feature test issues

Cool, we already have a task about proper teardown and a couple other
things in our backlog. we'll probably get to it in the next day or so. I'll
take a look at the other stuff.

Also, regarding speed, even without the app startup time, end to end tests
are always going to be relatively slow. We definitely want to make sure
that the time it takes to run the tests does not grow to where it is a
deterrent to running them.
There are a variety of things we can do to help address that as our suite
grows. For instance, we could consider parallelizing the tests, making
setup and teardown more efficient, combining related tests, or even
breaking the tests into suites and running only some of them locally by
default.
Since we only have a couple feature tests so far the speed hasn't really
felt like an issue for me yet, but I understand it may be different if you
are trying to run in a variety of configurations.

Out of curiosity, what is the goal in supporting multiple python versions?
Are we working on moving to 3.x and just haven't gotten fully there yet?

Tira

On Sun, Feb 26, 2017, 3:39 AM Dave Page <dpage(at)pgadmin(dot)org> wrote:

> Hi Tira, George,
>
> I've just been updating our internal automated test system to run the
> feature tests, and ran into a couple of additional issues that need to
> be addressed. Can you look into the following please?
>
> - When starting pgAdmin, it's using the default configuration database
> (CONFIG['SQLITE_PATH']), however for testing we should be using
> CONFIG['TEST_SQLITE_PATH']). This means that it's polluting the user's
> default configuration (and if they don't have one, causing an
> additional initialisation step).
>
> - With Python 2.6, the following failure is seen when the first
> feature test is run:
>
> Traceback (most recent call last):
> File
> "/var/lib/jenkins/workspace/pgadmin4-master-python26/web/regression/runtests.py",
> line 286, in <module>
> verbosity=2).run(suite)
> File
> "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/runner.py",
> line 172, in run
> test(result)
> File
> "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/suite.py",
> line 87, in __call__
> return self.run(*args, **kwds)
> File
> "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/suite.py",
> line 126, in run
> test(result)
> File
> "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/case.py",
> line 673, in __call__
> return self.run(*args, **kwds)
> File
> "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/case.py",
> line 633, in run
> self._feedErrorsToResult(result, outcome.errors)
> File
> "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/case.py",
> line 563, in _feedErrorsToResult
> if issubclass(exc_info[0], self.failureException):
> TypeError: issubclass() arg 2 must be a class or tuple of classes
>
> For completeness, other issues outstanding that we've previously discussed:
>
> - pgAdmin processes may remain running after test failures.
>
> - The test suite may hang following a feature test failure, at the end
> of the run.
>
> - The screenshot functionality should be fixed (ideally) or removed.
>
> - The tests really need to run with a single instantiation of pgAdmin.
> It's clearly going to be far too slow to start/stop pgAdmin for every
> test once we start adding more (and moving forward, I really want
> feature tests to become the default to ensure we're end-to-end testing
> everything). For reference, each test run (currently one version of
> Python, against 5 different database servers) is now taking ~5 minutes
> vs. 1m47s without the feature tests.
>
> On the plus side, test runs are now green across the board with
> feature tests enabled, except for Python 2.6 :-)
>
> Thanks!
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>