Re: Re: Server side cursor limitations for on demand loading of data in query tool [RM2137] [pgAdmin4]

Hi,

That's better - the failures are far less random now :-). I got the
following two though, on both PG and EPAS 9.5:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
Query tool feature test
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
line 95, in runTest
self._query_tool_explain_analyze_buffers()
File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
line 443, in _query_tool_explain_analyze_buffers
canvas.find_element_by_xpath("//*[contains(string(), 'Shared Read
Blocks')]")
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
line 260, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
line 508, in find_element
{"using": by, "value": value})['value']
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
line 491, in _execute
return self._parent.execute(command, params)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
line 238, in execute
self.error_handler.check_response(response)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
line 193, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: no such element: Unable to locate
element: {"method":"xpath","selector":"//*[contains(string(), 'Shared
Read Blocks')]"}
(Session info: chrome=58.0.3029.110)
(Driver info: chromedriver=2.29.461585
(0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
x86_64)

======================================================================
ERROR: runTest (pgadmin.feature_tests.view_data_dml_queries.CheckForViewDataTest)
Validate Insert, Update operations in View data with given test data
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
line 104, in runTest
self._add_row()
File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
line 255, in _add_row
self._update_cell(cell_xpath, config_data[str(idx)])
File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
line 164, in _update_cell
cell_el = self.page.find_by_xpath(xpath)
File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
line 122, in find_by_xpath
return self.wait_for_element(lambda driver:
driver.find_element_by_xpath(xpath))
File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
line 205, in wait_for_element
return self._wait_for("element to exist", element_if_it_exists)
File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
line 255, in _wait_for
"Timed out waiting for " + waiting_for_message)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/support/wait.py",
line 71, in until
value = method(self._driver)
File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
line 200, in element_if_it_exists
if element.is_displayed() and element.is_enabled():
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
line 358, in is_displayed
return self._execute(Command.IS_ELEMENT_DISPLAYED)['value']
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
line 491, in _execute
return self._parent.execute(command, params)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
line 238, in execute
self.error_handler.check_response(response)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
line 193, in check_response
raise exception_class(message, screen, stacktrace)
StaleElementReferenceException: Message: stale element reference:
element is not attached to the page document
(Session info: chrome=58.0.3029.110)
(Driver info: chromedriver=2.29.461585
(0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
x86_64)

On 9.4 everything passes. On 9.6, I get one failure:

======================================================================
FAIL: runTest (pgadmin.feature_tests.view_data_dml_queries.CheckForViewDataTest)
Validate Insert, Update operations in View data with given test data
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
line 105, in runTest
self._verify_row_data(True)
File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
line 282, in _verify_row_data
self.assertEquals(cells[idx], config_data[str(idx)][1])
AssertionError: u'[default]' != u'1'
- [default]
+ 1

Screenshots attached.

On Fri, Jun 16, 2017 at 7:54 AM, Harshal Dhumal
<harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
> Hi Dave,
>
> Please find attached patch where I have added timeout of 2 seconds before
> selecting object menu in connect to server function.
>
> Note: Apply this patch on top of previous patch for on demand loading
> feature.
>
>
> --
> Harshal Dhumal
> Sr. Software Engineer
>
> EnterpriseDB India: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
> On Fri, Jun 16, 2017 at 2:25 AM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>>
>> Sounds good, thanks.
>>
>> On Thu, Jun 15, 2017 at 9:54 PM, Harshal Dhumal
>> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> > Hi
>> >
>> > On Fri, Jun 16, 2017 at 2:07 AM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>> >>
>> >> Hi
>> >>
>> >> On Thu, Jun 15, 2017 at 9:30 PM, Harshal Dhumal
>> >> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> >> > Hi Dave,
>> >> >
>> >> > Please find attached updated patch.
>> >> >
>> >> > On Thu, Jun 15, 2017 at 3:58 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>> >> >>
>> >> >> Hi
>> >> >>
>> >> >> On Wed, Jun 14, 2017 at 11:36 PM, Harshal Dhumal
>> >> >> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> >> >>>
>> >> >>> Hi Dave,
>> >> >>>
>> >> >>> Please find rebased patch for RM2137.
>> >> >>
>> >> >>
>> >> >> Looking very good. The only issues I see are:
>> >> >>
>> >> >> - The row headers should auto-size such that they can display the
>> >> >> row
>> >> >> numbers if the last row was displayed. E.g. if there are 12345 rows
>> >> >> in
>> >> >> total, then the row header should be sized to display 5 digits.
>> >> >>
>> >> > Fixed.
>> >> >
>> >> >
>> >> >>
>> >> >> - The tests are comprehensive, which is awesome. However, every time
>> >> >> I
>> >> >> ran
>> >> >> them, at least one of the feature tests failed. Unfortunately, it
>> >> >> was a
>> >> >> different one each time. In the last two runs, I got:
>> >> >>
>> >> >>
>> >> >> ======================================================================
>> >> >> ERROR: runTest
>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >> >> Query tool feature test
>> >> >>
>> >> >> ----------------------------------------------------------------------
>> >> >> Traceback (most recent call last):
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/git/pgadmin4/web/regression/feature_utils/base_feature_test.py",
>> >> >> line 40, in setUp
>> >> >> self.before()
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >> >> line 40, in before
>> >> >> self._connects_to_server()
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >> >> line 144, in _connects_to_server
>> >> >> self.page.driver.find_element_by_link_text("Create"))\
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> >> line 319, in find_element_by_link_text
>> >> >> return self.find_element(by=By.LINK_TEXT, value=link_text)
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> >> line 756, in find_element
>> >> >> 'value': value})['value']
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> >> line 238, in execute
>> >> >> self.error_handler.check_response(response)
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >> >> line 193, in check_response
>> >> >> raise exception_class(message, screen, stacktrace)
>> >> >> NoSuchElementException: Message: no such element: Unable to locate
>> >> >> element: {"method":"link text","selector":"Create"}
>> >> >> (Session info: chrome=58.0.3029.110)
>> >> >> (Driver info: chromedriver=2.29.461585
>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> >> >> x86_64)
>> >> >>
>> >> > I checked code from _connects_to_server function which is common in
>> >> > all
>> >> > features test cases. I didn't find anything wrong with this. If there
>> >> > is
>> >> > a
>> >> > bug in this function then all feature test must fail.
>> >> > Let me know if you are getting failure consistently in
>> >> > _connects_to_server
>> >> > function.
>> >>
>> >> I wondered if that one is a race condition. Do we need a short delay
>> >> before clicking the Object menu? I have seen this occasionally before.
>> >
>> >
>> > OK. In that case let's try putting 1-2 second delay and observer
>> > behaviour.
>> > I'll send separate patch for this tomorrow as this is not related to on
>> > demand query result feature or its test cases.
>> >
>> >>
>> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> ======================================================================
>> >> >> ERROR: runTest
>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >> >> Query tool feature test
>> >> >>
>> >> >> ----------------------------------------------------------------------
>> >> >> Traceback (most recent call last):
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >> >> line 119, in runTest
>> >> >> self._query_tool_auto_rollback_enabled()
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >> >> line 697, in _query_tool_auto_rollback_enabled
>> >> >> '//div[contains(@class, "sql-editor-message") and
>> >> >> contains(string(),
>> >> >> "COMMIT")]'
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> >> line 295, in find_element_by_xpath
>> >> >> return self.find_element(by=By.XPATH, value=xpath)
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> >> line 756, in find_element
>> >> >> 'value': value})['value']
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> >> line 238, in execute
>> >> >> self.error_handler.check_response(response)
>> >> >> File
>> >> >>
>> >> >>
>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >> >> line 193, in check_response
>> >> >> raise exception_class(message, screen, stacktrace)
>> >> >> NoSuchElementException: Message: no such element: Unable to locate
>> >> >> element: {"method":"xpath","selector":"//div[contains(@class,
>> >> >> "sql-editor-message") and contains(string(), "COMMIT")]"}
>> >> >> (Session info: chrome=58.0.3029.110)
>> >> >> (Driver info: chromedriver=2.29.461585
>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> >> >> x86_64)
>> >> >>
>> >> > I have updated Auto rollback enabled test in this patch.
>> >> >
>> >> >>
>> >> >> Relevant screenshots attached.
>> >> >>
>> >> >> - Can you tidy up the regression output a little please? Instead of:
>> >> >>
>> >> >> -------
>> >> >> runTest
>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >> >> Query tool feature test ... On demand result set on scrolling...
>> >> >> OK.
>> >> >>
>> >> >> On demand result set on grid select all...
>> >> >> OK.
>> >> >>
>> >> >> On demand result set on column select all...
>> >> >> OK.
>> >> >>
>> >> >> Explain query...
>> >> >> OK.
>> >> >>
>> >> >> Explain query with verbose...
>> >> >> OK.
>> >> >>
>> >> >> Explain query with costs...
>> >> >> OK.
>> >> >>
>> >> >> Explain analyze query...
>> >> >> OK.
>> >> >>
>> >> >> Explain analyze query with buffers...
>> >> >> OK.
>> >> >>
>> >> >> Explain analyze query with timing...
>> >> >> OK.
>> >> >>
>> >> >> Auto commit disabled...
>> >> >> OK.
>> >> >>
>> >> >> Auto commit enabled...
>> >> >> OK.
>> >> >>
>> >> >> Auto rollback enabled...
>> >> >> ERROR
>> >> >> -------
>> >> >>
>> >> >> Something like:
>> >> >>
>> >> >> -------
>> >> >> runTest
>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >> >> Query tool feature test ...
>> >> >> On demand result set on scrolling... OK.
>> >> >> On demand result set on grid select all... OK.
>> >> >> On demand result set on column select all... OK.
>> >> >> Explain query... OK.
>> >> >> Explain query with verbose... OK.
>> >> >> Explain query with costs... OK.
>> >> >> Explain analyze query... OK.
>> >> >> Explain analyze query with buffers... OK.
>> >> >> Explain analyze query with timing... OK.
>> >> >> Auto commit disabled... OK.
>> >> >> Auto commit enabled... OK.
>> >> >> Auto rollback enabled... ERROR
>> >> >> --------
>> >> >>
>> >> > Fixed.
>> >> >
>> >> >
>> >> >>
>> >> >> Thanks!
>> >> >>
>> >> >> --
>> >> >> Dave Page
>> >> >> Blog: http://pgsnake.blogspot.com
>> >> >> Twitter: @pgsnake
>> >> >>
>> >> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >> >> The Enterprise PostgreSQL Company
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Dave Page
>> >> Blog: http://pgsnake.blogspot.com
>> >> Twitter: @pgsnake
>> >>
>> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >> The Enterprise PostgreSQL Company
>> >
>> >
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company