Re: pgAdmin4: Test result enhancement patch

Hi Dave,

Please find the revised patch for test result enhancement.

What's in the patch:
1. The test result summary will store in JSON file.
2. Removed some redundant code from regression/test_utils.py
3. Added the scenario names for feature tests.
4. To print test scenario names in failed and skipped test cases, I
override apply_scenario() function in regression/test_utils.py

On Mon, Apr 3, 2017 at 12:32 PM, Navnath Gadakh <
navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:

> Hi Akshay,
>
> Please find the revised patch for test result enhancement.
>
> *What's in the patch:*
> 1. The test result summary will store in JSON file.
> 2. Removed some redundant code from *regression/test_utils.py*
> *3. A*dded the scenario names for feature tests.
> 4. To print test scenario names in failed and skipped test cases, I o
> verride *apply_scenario()* function in *regression/test_utils.py*
>
> On Fri, Mar 31, 2017 at 6:16 PM, Akshay Joshi <
> akshay(dot)joshi(at)enterprisedb(dot)com> wrote:
>
>> Hi Navnath
>>
>> I have run the updated patch. It is working fine with Python 2.7 but I am
>> facing following error with Python 3.5, can you please look into it:
>>
>> ======================================================================
>>
>> ERROR: runTest (pgadmin.feature_tests.connect_to_server_feature_test.
>> ConnectsToServerFeatureTest)
>>
>> Test database connection which can be created from the UI
>>
>> ----------------------------------------------------------------------
>>
>> Traceback (most recent call last):
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/base_feature_test.py", line 33, in setUp
>>
>> self.page.reset_layout()
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/pgadmin_page.py", line 33, in reset_layout
>>
>> self.click_modal_ok()
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/pgadmin_page.py", line 38, in click_modal_ok
>>
>> self.click_element(self.find_by_xpath("//button[contains(.,'OK')]"))
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/pgadmin_page.py", line 71, in find_by_xpath
>>
>> return self.wait_for_element(lambda driver: driver.
>> find_element_by_xpath(xpath))
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/pgadmin_page.py", line 128, in wait_for_element
>>
>> return self._wait_for("element to exist", element_if_it_exists)
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/pgadmin_page.py", line 162, in _wait_for
>>
>> "Timed out waiting for " + waiting_for_message)
>>
>> File "/Users/akshay/Development/Workspace/lib/python3.5/site-pack
>> ages/selenium/webdriver/support/wait.py", line 80, in until
>>
>> raise TimeoutException(message, screen, stacktrace)
>>
>> selenium.common.exceptions.TimeoutException: Message: Timed out waiting
>> for element to exist
>>
>>
>>
>> ======================================================================
>>
>> ERROR: runTest (pgadmin.feature_tests.table_ddl_feature_test.
>> TableDdlFeatureTest)
>>
>> Test scenarios for acceptance tests
>>
>> ----------------------------------------------------------------------
>>
>> Traceback (most recent call last):
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/base_feature_test.py", line 33, in setUp
>>
>> self.page.reset_layout()
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/pgadmin_page.py", line 31, in reset_layout
>>
>> self.click_element(self.find_by_partial_link_text("File"))
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/pgadmin_page.py", line 90, in click_element
>>
>> return self._wait_for("clicking the element not to throw an
>> exception", click_succeeded)
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/feature_
>> utils/pgadmin_page.py", line 162, in _wait_for
>>
>> "Timed out waiting for " + waiting_for_message)
>>
>> File "/Users/akshay/Development/Workspace/lib/python3.5/site-pack
>> ages/selenium/webdriver/support/wait.py", line 80, in until
>>
>> raise TimeoutException(message, screen, stacktrace)
>>
>> selenium.common.exceptions.TimeoutException: Message: Timed out waiting
>> for clicking the element not to throw an exception
>>
>>
>>
>> ----------------------------------------------------------------------
>>
>> Ran 153 tests in 45.493s
>>
>>
>> FAILED (errors=2, skipped=16)
>>
>>
>> ======================================================================
>>
>> Test Result Summary
>>
>> ======================================================================
>>
>>
>> Traceback (most recent call last):
>>
>> File "runtests.py", line 354, in <module>
>>
>> skipped_cases)
>>
>> File "/Users/akshay/Development/pgadmin4/web/regression/python_
>> test_utils/test_utils.py", line 442, in get_scenario_name
>>
>> key, value = case_name_dict.items()[0]
>>
>> TypeError: 'dict_items' object does not support indexing
>>
>> Resolved.
>
> Thanks!
>
> On Thu, Mar 30, 2017 at 8:04 PM, Navnath Gadakh <
>> navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
>>
>>> Hi Dave,
>>>
>>> Please find the revised patch for test result enhancement.
>>>
>>> *What's in the patch:*
>>> 1. The test result summary will store in JSON file.
>>> 2. Removed some redundant code from *regression/test_utils.py*
>>> *3. A*dded the scenario names for feature tests.
>>> 4. To print test scenario names in failed and skipped test cases, I o
>>> verride *apply_scenario()* function in *regression/test_utils.py*
>>>
>>> I have also attached the sample JSON file with the test result as per
>>> your suggestions.
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Wed, Mar 29, 2017 at 6:03 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>>>
>>>> On Wed, Mar 29, 2017 at 4:12 AM, Navnath Gadakh
>>>> <navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
>>>> > Hi,
>>>> >
>>>> > On Mon, Mar 27, 2017 at 5:37 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>>>> >>
>>>> >> Hi
>>>> >>
>>>> >> On Mon, Mar 27, 2017 at 12:15 AM, Navnath Gadakh
>>>> >> <navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
>>>> >> > Hello Dave,
>>>> >> >
>>>> >> > On Fri, Mar 24, 2017 at 9:10 PM, Dave Page <dpage(at)pgadmin(dot)org>
>>>> wrote:
>>>> >> >>
>>>> >> >> Hi
>>>> >> >>
>>>> >> >> On Fri, Mar 24, 2017 at 3:13 PM, Navnath Gadakh
>>>> >> >> <navnath(dot)gadakh(at)enterprisedb(dot)com> wrote:
>>>> >> >> >
>>>> >> >> >> When running with the patch:
>>>> >> >> >>
>>>> >> >> >> 1) The browser isn't closed, and the script never exits - it
>>>> just
>>>> >> >> >> sits
>>>> >> >> >> indefinitely at:
>>>> >> >> >>
>>>> >> >> >> =====
>>>> >> >> >> Please check output in file:
>>>> >> >> >> /Users/dpage/git/pgadmin4/web/regression/regression.log
>>>> >> >> >>
>>>> >> >> >> make: *** [check] Error 1
>>>> >> >> >> =====
>>>> >> >> >>
>>>> >> >> >> without returning to a shell prompt. The browser exits when I
>>>> hit
>>>> >> >> >> Ctrl+C.
>>>> >> >>
>>>> >> >> The above is still a problem. In fact, not only do I have to hit
>>>> >> >> Ctrl+C, but then the browser prompts me to check I really do want
>>>> to
>>>> >> >> exit.
>>>> >> >>
>>>> >> >> There's also another problem that just showed up. I got the
>>>> following
>>>> >> >> failure on PG 9.4 (due to a known intermittent bug that Ashesh
>>>> and
>>>> >> >> Tira(at)Pivotal are working on). Note how it's not reported in the
>>>> >> >> summary (or the JSON output):
>>>> >> >
>>>> >> >
>>>> >> > I found the issue, In the feature tests we need to add a scenario
>>>> name
>>>> >> > for
>>>> >> > each test case. the purpose of this patch is to print the
>>>> failed/skipped
>>>> >> > test class name with the scenario name like:
>>>> >> >
>>>> >> > 152 tests passed
>>>> >> >
>>>> >> > 1 test failed:
>>>> >> >
>>>> >> > LoginRoleGetTestCase (Check Role Node)
>>>> >> >
>>>> >> > 16 tests skipped:
>>>> >> >
>>>> >> > SynonymGetTestCase (Fetch synonym Node URL)
>>>> >> >
>>>> >> > But our in-built test framework does not provide that scenario
>>>> name with
>>>> >> > failed/skipped test case that's why I override apply_scenario()
>>>> >> > function.
>>>> >> >
>>>> >> > def apply_scenario(scenario, test):
>>>> >> >
>>>> >> > name, parameters = scenario
>>>> >> >
>>>> >> > parameters["scenario_name"] = name
>>>> >> >
>>>> >> > While printing the result, I have checked the if 'scenario_name'
>>>> in test
>>>> >> > as
>>>> >> > we need to print scenario name in test summary as well as in JSON
>>>> file.
>>>> >> >
>>>> >> > I can do it without scenario name but for better understanding
>>>> which
>>>> >> > test
>>>> >> > scenario is failed it's good to add a scenario name with each
>>>> test case.
>>>> >>
>>>> >> OK.
>>>> >>
>>>> >> > See this is how test cases looks like while printing on console
>>>> >> >
>>>> >> > API:
>>>> >> >
>>>> >> > runTest
>>>> >> >
>>>> >> > (pgadmin.browser.server_groups.servers.databases.schemas.typ
>>>> es.tests.test_types_put.TypesUpdateTestCase)
>>>> >> >
>>>> >> > Update type under schema node ... ok
>>>> >> >
>>>> >> > Feature tests:
>>>> >> >
>>>> >> > runTest
>>>> >> >
>>>> >> > (pgadmin.utils.tests.test_versioned_template_loader.TestVers
>>>> ionedTemplateLoader)
>>>> >> > ... ok
>>>> >> >
>>>> >> > No scenario name in feature tests.
>>>> >> >
>>>> >>
>>>> >> OK, is that easy to fix while you're at it?
>>>> >
>>>> >
>>>> > I have two solutions-
>>>> >
>>>> > 1. Need a little hack to skip scenario/test name if that does not
>>>> exist, but
>>>> > that's not the best idea.
>>>> >
>>>> > 2. Owner of feature tests should add scenario/test name to each
>>>> feature
>>>> > test. In the summary also we will know for which scenario test is
>>>> failing or
>>>> > skipping.
>>>> > This is ideal and long term solution and I prefer it.
>>>>
>>>> Agreed - and as there are only 2 feature tests, you should be able to
>>>> fix them up pretty quickly :-p
>>>>
>>>> Once code is in the repo, it's "ours", meaning the entire communities.
>>>> I wouldn't expect us to ping all issues back to Pivotal - we're one
>>>> team on this.
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Navnath Gadakh
>>>
>>> EnterpriseDB Corporation
>>> The Enterprise PostgreSQL Company
>>>
>>>
>>>
>>>
>>> --
>>> Sent via pgadmin-hackers mailing list (pgadmin-hackers(at)postgresql(dot)org)
>>> To make changes to your subscription:
>>> http://www.postgresql.org/mailpref/pgadmin-hackers
>>>
>>>
>>
>>
>> --
>> *Akshay Joshi*
>> *Principal Software Engineer *
>>
>>
>>
>> *Phone: +91 20-3058-9517Mobile: +91 976-788-8246*
>>
>
>
>
> --
> Regards,
> Navnath Gadakh
>
> EnterpriseDB Corporation
> The Enterprise PostgreSQL Company
>
>
>

--
Regards,
Navnath Gadakh

EnterpriseDB Corporation
The Enterprise PostgreSQL Company