Re: pgAdmin4: Test result enhancement patch

Hi Akshay,

Please find the revised patch for test result enhancement.

*What's in the patch:*
1. The test result summary will store in JSON file.
2. Removed some redundant code from *regression/test_utils.py*
*3. A*dded the scenario names for feature tests.
4. To print test scenario names in failed and skipped test cases, I o
verride *apply_scenario()* function in *regression/test_utils.py*

On Fri, Mar 31, 2017 at 6:16 PM, Akshay Joshi <akshay(dot)joshi(at)enterprisedb(dot)com
> wrote:

> Hi Navnath
>
> I have run the updated patch. It is working fine with Python 2.7 but I am
> facing following error with Python 3.5, can you please look into it:
>
> ======================================================================
>
> ERROR: runTest (pgadmin.feature_tests.connect_to_server_feature_test.
> ConnectsToServerFeatureTest)
>
> Test database connection which can be created from the UI
>
> ----------------------------------------------------------------------
>
> Traceback (most recent call last):
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/base_feature_test.py", line 33, in setUp
>
> self.page.reset_layout()
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/pgadmin_page.py", line 33, in reset_layout
>
> self.click_modal_ok()
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/pgadmin_page.py", line 38, in click_modal_ok
>
> self.click_element(self.find_by_xpath("//button[contains(.,'OK')]"))
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/pgadmin_page.py", line 71, in find_by_xpath
>
> return self.wait_for_element(lambda driver: driver.
> find_element_by_xpath(xpath))
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/pgadmin_page.py", line 128, in wait_for_element
>
> return self._wait_for("element to exist", element_if_it_exists)
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/pgadmin_page.py", line 162, in _wait_for
>
> "Timed out waiting for " + waiting_for_message)
>
> File "/Users/akshay/Development/Workspace/lib/python3.5/site-
> packages/selenium/webdriver/support/wait.py", line 80, in until
>
> raise TimeoutException(message, screen, stacktrace)
>
> selenium.common.exceptions.TimeoutException: Message: Timed out waiting
> for element to exist
>
>
>
> ======================================================================
>
> ERROR: runTest (pgadmin.feature_tests.table_ddl_feature_test.
> TableDdlFeatureTest)
>
> Test scenarios for acceptance tests
>
> ----------------------------------------------------------------------
>
> Traceback (most recent call last):
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/base_feature_test.py", line 33, in setUp
>
> self.page.reset_layout()
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/pgadmin_page.py", line 31, in reset_layout
>
> self.click_element(self.find_by_partial_link_text("File"))
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/pgadmin_page.py", line 90, in click_element
>
> return self._wait_for("clicking the element not to throw an
> exception", click_succeeded)
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> feature_utils/pgadmin_page.py", line 162, in _wait_for
>
> "Timed out waiting for " + waiting_for_message)
>
> File "/Users/akshay/Development/Workspace/lib/python3.5/site-
> packages/selenium/webdriver/support/wait.py", line 80, in until
>
> raise TimeoutException(message, screen, stacktrace)
>
> selenium.common.exceptions.TimeoutException: Message: Timed out waiting
> for clicking the element not to throw an exception
>
>
>
> ----------------------------------------------------------------------
>
> Ran 153 tests in 45.493s
>
>
> FAILED (errors=2, skipped=16)
>
>
> ======================================================================
>
> Test Result Summary
>
> ======================================================================
>
>
> Traceback (most recent call last):
>
> File "runtests.py", line 354, in <module>
>
> skipped_cases)
>
> File "/Users/akshay/Development/pgadmin4/web/regression/
> python_test_utils/test_utils.py", line 442, in get_scenario_name
>
> key, value = case_name_dict.items()[0]
>
> TypeError: 'dict_items' object does not support indexing
>
> Resolved.

Thanks!

On Thu, Mar 30, 2017 at 8:04 PM, Navnath Gadakh <
> navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
>
>> Hi Dave,
>>
>> Please find the revised patch for test result enhancement.
>>
>> *What's in the patch:*
>> 1. The test result summary will store in JSON file.
>> 2. Removed some redundant code from *regression/test_utils.py*
>> *3. A*dded the scenario names for feature tests.
>> 4. To print test scenario names in failed and skipped test cases, I o
>> verride *apply_scenario()* function in *regression/test_utils.py*
>>
>> I have also attached the sample JSON file with the test result as per
>> your suggestions.
>>
>> Thanks!
>>
>>
>>
>> On Wed, Mar 29, 2017 at 6:03 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>>
>>> On Wed, Mar 29, 2017 at 4:12 AM, Navnath Gadakh
>>> <navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
>>> > Hi,
>>> >
>>> > On Mon, Mar 27, 2017 at 5:37 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>>> >>
>>> >> Hi
>>> >>
>>> >> On Mon, Mar 27, 2017 at 12:15 AM, Navnath Gadakh
>>> >> <navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
>>> >> > Hello Dave,
>>> >> >
>>> >> > On Fri, Mar 24, 2017 at 9:10 PM, Dave Page <dpage(at)pgadmin(dot)org>
>>> wrote:
>>> >> >>
>>> >> >> Hi
>>> >> >>
>>> >> >> On Fri, Mar 24, 2017 at 3:13 PM, Navnath Gadakh
>>> >> >> <navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
>>> >> >> >
>>> >> >> >> When running with the patch:
>>> >> >> >>
>>> >> >> >> 1) The browser isn't closed, and the script never exits - it
>>> just
>>> >> >> >> sits
>>> >> >> >> indefinitely at:
>>> >> >> >>
>>> >> >> >> =====
>>> >> >> >> Please check output in file:
>>> >> >> >> /Users/dpage/git/pgadmin4/web/regression/regression.log
>>> >> >> >>
>>> >> >> >> make: *** [check] Error 1
>>> >> >> >> =====
>>> >> >> >>
>>> >> >> >> without returning to a shell prompt. The browser exits when I
>>> hit
>>> >> >> >> Ctrl+C.
>>> >> >>
>>> >> >> The above is still a problem. In fact, not only do I have to hit
>>> >> >> Ctrl+C, but then the browser prompts me to check I really do want
>>> to
>>> >> >> exit.
>>> >> >>
>>> >> >> There's also another problem that just showed up. I got the
>>> following
>>> >> >> failure on PG 9.4 (due to a known intermittent bug that Ashesh and
>>> >> >> Tira(at)Pivotal are working on). Note how it's not reported in the
>>> >> >> summary (or the JSON output):
>>> >> >
>>> >> >
>>> >> > I found the issue, In the feature tests we need to add a scenario
>>> name
>>> >> > for
>>> >> > each test case. the purpose of this patch is to print the
>>> failed/skipped
>>> >> > test class name with the scenario name like:
>>> >> >
>>> >> > 152 tests passed
>>> >> >
>>> >> > 1 test failed:
>>> >> >
>>> >> > LoginRoleGetTestCase (Check Role Node)
>>> >> >
>>> >> > 16 tests skipped:
>>> >> >
>>> >> > SynonymGetTestCase (Fetch synonym Node URL)
>>> >> >
>>> >> > But our in-built test framework does not provide that scenario
>>> name with
>>> >> > failed/skipped test case that's why I override apply_scenario()
>>> >> > function.
>>> >> >
>>> >> > def apply_scenario(scenario, test):
>>> >> >
>>> >> > name, parameters = scenario
>>> >> >
>>> >> > parameters["scenario_name"] = name
>>> >> >
>>> >> > While printing the result, I have checked the if 'scenario_name' in
>>> test
>>> >> > as
>>> >> > we need to print scenario name in test summary as well as in JSON
>>> file.
>>> >> >
>>> >> > I can do it without scenario name but for better understanding which
>>> >> > test
>>> >> > scenario is failed it's good to add a scenario name with each test
>>> case.
>>> >>
>>> >> OK.
>>> >>
>>> >> > See this is how test cases looks like while printing on console
>>> >> >
>>> >> > API:
>>> >> >
>>> >> > runTest
>>> >> >
>>> >> > (pgadmin.browser.server_groups.servers.databases.schemas.typ
>>> es.tests.test_types_put.TypesUpdateTestCase)
>>> >> >
>>> >> > Update type under schema node ... ok
>>> >> >
>>> >> > Feature tests:
>>> >> >
>>> >> > runTest
>>> >> >
>>> >> > (pgadmin.utils.tests.test_versioned_template_loader.TestVers
>>> ionedTemplateLoader)
>>> >> > ... ok
>>> >> >
>>> >> > No scenario name in feature tests.
>>> >> >
>>> >>
>>> >> OK, is that easy to fix while you're at it?
>>> >
>>> >
>>> > I have two solutions-
>>> >
>>> > 1. Need a little hack to skip scenario/test name if that does not
>>> exist, but
>>> > that's not the best idea.
>>> >
>>> > 2. Owner of feature tests should add scenario/test name to each feature
>>> > test. In the summary also we will know for which scenario test is
>>> failing or
>>> > skipping.
>>> > This is ideal and long term solution and I prefer it.
>>>
>>> Agreed - and as there are only 2 feature tests, you should be able to
>>> fix them up pretty quickly :-p
>>>
>>> Once code is in the repo, it's "ours", meaning the entire communities.
>>> I wouldn't expect us to ping all issues back to Pivotal - we're one
>>> team on this.
>>>
>>> Thanks!
>>>
>>> --
>>> Dave Page
>>> Blog: http://pgsnake.blogspot.com
>>> Twitter: @pgsnake
>>>
>>> EnterpriseDB UK: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>
>>
>>
>> --
>> Regards,
>> Navnath Gadakh
>>
>> EnterpriseDB Corporation
>> The Enterprise PostgreSQL Company
>>
>>
>>
>>
>> --
>> Sent via pgadmin-hackers mailing list (pgadmin-hackers(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgadmin-hackers
>>
>>
>
>
> --
> *Akshay Joshi*
> *Principal Software Engineer *
>
>
>
> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246*
>

--
Regards,
Navnath Gadakh

EnterpriseDB Corporation
The Enterprise PostgreSQL Company