Re: pgAdmin4: Test result enhancement patch

Hello Dave,

On Fri, Mar 24, 2017 at 9:10 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:

> Hi
>
> On Fri, Mar 24, 2017 at 3:13 PM, Navnath Gadakh
> <navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
> >
> >> When running with the patch:
> >>
> >> 1) The browser isn't closed, and the script never exits - it just sits
> >> indefinitely at:
> >>
> >> =====
> >> Please check output in file:
> >> /Users/dpage/git/pgadmin4/web/regression/regression.log
> >>
> >> make: *** [check] Error 1
> >> =====
> >>
> >> without returning to a shell prompt. The browser exits when I hit
> Ctrl+C.
>
> The above is still a problem. In fact, not only do I have to hit
> Ctrl+C, but then the browser prompts me to check I really do want to
> exit.
>
> There's also another problem that just showed up. I got the following
> failure on PG 9.4 (due to a known intermittent bug that Ashesh and
> Tira(at)Pivotal are working on). Note how it's not reported in the
> summary (or the JSON output):
>

I found the issue, In the feature tests we need to add a scenario name for
each test case. the purpose of this patch is to print the *failed/skipped* test
class name with the scenario name like:

*152 tests passed*

*1 test failed:*

* LoginRoleGetTestCase (Check Role Node)*

*16 tests skipped:*

* SynonymGetTestCase (Fetch synonym Node URL)*

But our in-built test framework does not provide that scenario name with
failed/skipped test case that's why I override *apply_scenario() *function.

*def apply_scenario(scenario, test):*

* name, parameters = scenario*

* parameters["scenario_name"] = name*

While printing the result, I have checked the *if 'scenario_name' in
test as we need to print scenario name in test summary as well as in JSON
file.*

I can do it without scenario name but for better understanding which test
scenario is failed it's good to add a scenario name with each test case.

See this is how test cases looks like while printing on console

*API:*

*runTest
(pgadmin.browser.server_groups.servers.databases.schemas.types.tests.test_types_put.TypesUpdateTestCase)*

*Update type under schema node ... ok*

* Feature tests:*

*runTest (pgadmin.utils.tests.test_versioned_template_loader.TestVersionedTemplateLoader)...
ok*

No scenario name in feature tests.

but our inbuilt test freamework does not provide

>
> runTest (pgadmin.feature_tests.connect_to_server_feature_test.
> ConnectsToServerFeatureTest)
> ... ERROR
> runTest (pgadmin.feature_tests.table_ddl_feature_test.TableDdlFeatureTest)
> ... ok
> runTest (pgadmin.utils.tests.test_versioned_template_loader.
> TestVersionedTemplateLoader)
> ... ok
>
> ======================================================================
> ERROR: runTest (pgadmin.feature_tests.connect_to_server_feature_test.
> ConnectsToServerFeatureTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> connect_to_server_feature_test.py",
> line 37, in runTest
> self._tables_node_expandable()
> File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> connect_to_server_feature_test.py",
> line 73, in _tables_node_expandable
> self.page.toggle_open_tree_item('test_table')
> File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> pgadmin_page.py",
> line 68, in toggle_open_tree_item
> self.find_by_xpath("//*[(at)id='tree']//*[.='" + tree_item_text +
> "']/../*[(at)class='aciTreeButton']").click()
> File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> pgadmin_page.py",
> line 71, in find_by_xpath
> return self.wait_for_element(lambda driver:
> driver.find_element_by_xpath(xpath))
> File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> pgadmin_page.py",
> line 128, in wait_for_element
> return self._wait_for("element to exist", element_if_it_exists)
> File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> pgadmin_page.py",
> line 162, in _wait_for
> "Timed out waiting for " + waiting_for_message)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/support/wait.py",
> line 80, in until
> raise TimeoutException(message, screen, stacktrace)
> TimeoutException: Message: Timed out waiting for element to exist
>
>
> ----------------------------------------------------------------------
> Ran 153 tests in 60.698s
>
> FAILED (errors=1, skipped=12)
>
> ...
> ...
> ...
>
This error will be in shown in the summary as well as in JSON file when we
add 'scenario_name' for each feature test.

If you have any other thinking please let me know.

>
> ======================================================================
> Test Result Summary
> ======================================================================
>
> Regression - EPAS 9.5:
>
> 153 tests passed
> 0 tests failed
> 0 tests skipped
>
> Regression - PG 9.5:
>
> 141 tests passed
> 0 tests failed
> 12 tests skipped:
> SynonymGetTestCase (Fetch synonym Node URL)
> PackageDeleteTestCase (Fetch Package Node URL)
> ResourceGroupsGetTestCase (Get resource groups)
> SynonymDeleteTestCase (Fetch synonym Node URL)
> ResourceGroupsAddTestCase (Add resource groups)
> PackagePutTestCase (Fetch Package Node URL)
> SynonymPutTestCase (Fetch synonym Node URL)
> ResourceGroupsPutTestCase (Put resource groups)
> ResourceGroupsDeleteTestCase (Delete resource groups)
> SynonymAddTestCase (Default Node URL)
> PackageAddTestCase (Fetch Package Node URL)
> PackageGetTestCase (Fetch Package Node URL)
>
> Regression - PG 9.4:
>
> 141 tests passed
> 0 tests failed
> 12 tests skipped:
> SynonymGetTestCase (Fetch synonym Node URL)
> PackageDeleteTestCase (Fetch Package Node URL)
> ResourceGroupsGetTestCase (Get resource groups)
> SynonymDeleteTestCase (Fetch synonym Node URL)
> ResourceGroupsAddTestCase (Add resource groups)
> PackagePutTestCase (Fetch Package Node URL)
> SynonymPutTestCase (Fetch synonym Node URL)
> ResourceGroupsPutTestCase (Put resource groups)
> ResourceGroupsDeleteTestCase (Delete resource groups)
> SynonymAddTestCase (Default Node URL)
> PackageAddTestCase (Fetch Package Node URL)
> PackageGetTestCase (Fetch Package Node URL)
>
> ======================================================================
>
Thanks.

>
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

--
Regards,
Navnath Gadakh

EnterpriseDB Corporation
The Enterprise PostgreSQL Company