Re: [pgadmin-hackers] Re: Server side cursor limitations for on demand loading of data in query tool [RM2137] [pgAdmin4]

Hi Dave,

Please find updated rebased patch for RM2137

On Fri, Jun 23, 2017 at 9:00 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:

> Hi Harshal,
>
> When can we expect an updated version of this patch? I think it's
> important to get this into the next release.
>
> Thanks!
>
> On Fri, Jun 16, 2017 at 10:55 AM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
> > Hi,
> >
> > That's better - the failures are far less random now :-). I got the
> > following two though, on both PG and EPAS 9.5:
> >
> > ======================================================================
> > ERROR: runTest (pgadmin.feature_tests.query_tool_tests.
> QueryToolFeatureTest)
> > Query tool feature test
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> > File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> query_tool_tests.py",
> > line 95, in runTest
> > self._query_tool_explain_analyze_buffers()
> > File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> query_tool_tests.py",
> > line 443, in _query_tool_explain_analyze_buffers
> > canvas.find_element_by_xpath("//*[contains(string(), 'Shared Read
> > Blocks')]")
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webelement.py",
> > line 260, in find_element_by_xpath
> > return self.find_element(by=By.XPATH, value=xpath)
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webelement.py",
> > line 508, in find_element
> > {"using": by, "value": value})['value']
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webelement.py",
> > line 491, in _execute
> > return self._parent.execute(command, params)
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webdriver.py",
> > line 238, in execute
> > self.error_handler.check_response(response)
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/errorhandler.py",
> > line 193, in check_response
> > raise exception_class(message, screen, stacktrace)
> > NoSuchElementException: Message: no such element: Unable to locate
> > element: {"method":"xpath","selector":"//*[contains(string(), 'Shared
> > Read Blocks')]"}
> > (Session info: chrome=58.0.3029.110)
> > (Driver info: chromedriver=2.29.461585
> > (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
> > x86_64)
> >
> >
> > ======================================================================
> > ERROR: runTest (pgadmin.feature_tests.view_data_dml_queries.
> CheckForViewDataTest)
> > Validate Insert, Update operations in View data with given test data
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> > File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> view_data_dml_queries.py",
> > line 104, in runTest
> > self._add_row()
> > File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> view_data_dml_queries.py",
> > line 255, in _add_row
> > self._update_cell(cell_xpath, config_data[str(idx)])
> > File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> view_data_dml_queries.py",
> > line 164, in _update_cell
> > cell_el = self.page.find_by_xpath(xpath)
> > File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> pgadmin_page.py",
> > line 122, in find_by_xpath
> > return self.wait_for_element(lambda driver:
> > driver.find_element_by_xpath(xpath))
> > File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> pgadmin_page.py",
> > line 205, in wait_for_element
> > return self._wait_for("element to exist", element_if_it_exists)
> > File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> pgadmin_page.py",
> > line 255, in _wait_for
> > "Timed out waiting for " + waiting_for_message)
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/support/wait.py",
> > line 71, in until
> > value = method(self._driver)
> > File "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> pgadmin_page.py",
> > line 200, in element_if_it_exists
> > if element.is_displayed() and element.is_enabled():
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webelement.py",
> > line 358, in is_displayed
> > return self._execute(Command.IS_ELEMENT_DISPLAYED)['value']
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webelement.py",
> > line 491, in _execute
> > return self._parent.execute(command, params)
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webdriver.py",
> > line 238, in execute
> > self.error_handler.check_response(response)
> > File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/errorhandler.py",
> > line 193, in check_response
> > raise exception_class(message, screen, stacktrace)
> > StaleElementReferenceException: Message: stale element reference:
> > element is not attached to the page document
> > (Session info: chrome=58.0.3029.110)
> > (Driver info: chromedriver=2.29.461585
> > (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
> > x86_64)
> >
>

I checked my feature test cases for any database server version/type (PG,
EPAS) specific failures and I found that test cases are working fine on all
of them. However I have slightly modified test case to overcome above
mentioned failures.

> >
> > On 9.4 everything passes. On 9.6, I get one failure:
> >
> > ======================================================================
> > FAIL: runTest (pgadmin.feature_tests.view_data_dml_queries.
> CheckForViewDataTest)
> > Validate Insert, Update operations in View data with given test data
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> > File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> view_data_dml_queries.py",
> > line 105, in runTest
> > self._verify_row_data(True)
> > File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> view_data_dml_queries.py",
> > line 282, in _verify_row_data
> > self.assertEquals(cells[idx], config_data[str(idx)][1])
> > AssertionError: u'[default]' != u'1'
> > - [default]
> > + 1
> >
>
I guess Khushboo has fixed this recently.

> > Screenshots attached.
> >
> > On Fri, Jun 16, 2017 at 7:54 AM, Harshal Dhumal
> > <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
> >> Hi Dave,
> >>
> >> Please find attached patch where I have added timeout of 2 seconds
> before
> >> selecting object menu in connect to server function.
> >>
> >> Note: Apply this patch on top of previous patch for on demand loading
> >> feature.
> >>
> >>
> >> --
> >> Harshal Dhumal
> >> Sr. Software Engineer
> >>
> >> EnterpriseDB India: http://www.enterprisedb.com
> >> The Enterprise PostgreSQL Company
> >>
> >> On Fri, Jun 16, 2017 at 2:25 AM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
> >>>
> >>> Sounds good, thanks.
> >>>
> >>> On Thu, Jun 15, 2017 at 9:54 PM, Harshal Dhumal
> >>> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
> >>> > Hi
> >>> >
> >>> > On Fri, Jun 16, 2017 at 2:07 AM, Dave Page <dpage(at)pgadmin(dot)org>
> wrote:
> >>> >>
> >>> >> Hi
> >>> >>
> >>> >> On Thu, Jun 15, 2017 at 9:30 PM, Harshal Dhumal
> >>> >> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
> >>> >> > Hi Dave,
> >>> >> >
> >>> >> > Please find attached updated patch.
> >>> >> >
> >>> >> > On Thu, Jun 15, 2017 at 3:58 PM, Dave Page <dpage(at)pgadmin(dot)org>
> wrote:
> >>> >> >>
> >>> >> >> Hi
> >>> >> >>
> >>> >> >> On Wed, Jun 14, 2017 at 11:36 PM, Harshal Dhumal
> >>> >> >> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
> >>> >> >>>
> >>> >> >>> Hi Dave,
> >>> >> >>>
> >>> >> >>> Please find rebased patch for RM2137.
> >>> >> >>
> >>> >> >>
> >>> >> >> Looking very good. The only issues I see are:
> >>> >> >>
> >>> >> >> - The row headers should auto-size such that they can display the
> >>> >> >> row
> >>> >> >> numbers if the last row was displayed. E.g. if there are 12345
> rows
> >>> >> >> in
> >>> >> >> total, then the row header should be sized to display 5 digits.
> >>> >> >>
> >>> >> > Fixed.
> >>> >> >
> >>> >> >
> >>> >> >>
> >>> >> >> - The tests are comprehensive, which is awesome. However, every
> time
> >>> >> >> I
> >>> >> >> ran
> >>> >> >> them, at least one of the feature tests failed. Unfortunately, it
> >>> >> >> was a
> >>> >> >> different one each time. In the last two runs, I got:
> >>> >> >>
> >>> >> >>
> >>> >> >> ============================================================
> ==========
> >>> >> >> ERROR: runTest
> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
> >>> >> >> Query tool feature test
> >>> >> >>
> >>> >> >> ------------------------------------------------------------
> ----------
> >>> >> >> Traceback (most recent call last):
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/git/pgadmin4/web/regression/feature_utils/
> base_feature_test.py",
> >>> >> >> line 40, in setUp
> >>> >> >> self.before()
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> query_tool_tests.py",
> >>> >> >> line 40, in before
> >>> >> >> self._connects_to_server()
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> query_tool_tests.py",
> >>> >> >> line 144, in _connects_to_server
> >>> >> >> self.page.driver.find_element_by_link_text("Create"))\
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webdriver.py",
> >>> >> >> line 319, in find_element_by_link_text
> >>> >> >> return self.find_element(by=By.LINK_TEXT, value=link_text)
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webdriver.py",
> >>> >> >> line 756, in find_element
> >>> >> >> 'value': value})['value']
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webdriver.py",
> >>> >> >> line 238, in execute
> >>> >> >> self.error_handler.check_response(response)
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/errorhandler.py",
> >>> >> >> line 193, in check_response
> >>> >> >> raise exception_class(message, screen, stacktrace)
> >>> >> >> NoSuchElementException: Message: no such element: Unable to
> locate
> >>> >> >> element: {"method":"link text","selector":"Create"}
> >>> >> >> (Session info: chrome=58.0.3029.110)
> >>> >> >> (Driver info: chromedriver=2.29.461585
> >>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X
> 10.12.3
> >>> >> >> x86_64)
> >>> >> >>
> >>> >> > I checked code from _connects_to_server function which is common
> in
> >>> >> > all
> >>> >> > features test cases. I didn't find anything wrong with this. If
> there
> >>> >> > is
> >>> >> > a
> >>> >> > bug in this function then all feature test must fail.
> >>> >> > Let me know if you are getting failure consistently in
> >>> >> > _connects_to_server
> >>> >> > function.
> >>> >>
> >>> >> I wondered if that one is a race condition. Do we need a short delay
> >>> >> before clicking the Object menu? I have seen this occasionally
> before.
> >>> >
> >>> >
> >>> > OK. In that case let's try putting 1-2 second delay and observer
> >>> > behaviour.
> >>> > I'll send separate patch for this tomorrow as this is not related to
> on
> >>> > demand query result feature or its test cases.
> >>> >
> >>> >>
> >>> >>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> ============================================================
> ==========
> >>> >> >> ERROR: runTest
> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
> >>> >> >> Query tool feature test
> >>> >> >>
> >>> >> >> ------------------------------------------------------------
> ----------
> >>> >> >> Traceback (most recent call last):
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> query_tool_tests.py",
> >>> >> >> line 119, in runTest
> >>> >> >> self._query_tool_auto_rollback_enabled()
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/
> query_tool_tests.py",
> >>> >> >> line 697, in _query_tool_auto_rollback_enabled
> >>> >> >> '//div[contains(@class, "sql-editor-message") and
> >>> >> >> contains(string(),
> >>> >> >> "COMMIT")]'
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webdriver.py",
> >>> >> >> line 295, in find_element_by_xpath
> >>> >> >> return self.find_element(by=By.XPATH, value=xpath)
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webdriver.py",
> >>> >> >> line 756, in find_element
> >>> >> >> 'value': value})['value']
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/webdriver.py",
> >>> >> >> line 238, in execute
> >>> >> >> self.error_handler.check_response(response)
> >>> >> >> File
> >>> >> >>
> >>> >> >>
> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-
> packages/selenium/webdriver/remote/errorhandler.py",
> >>> >> >> line 193, in check_response
> >>> >> >> raise exception_class(message, screen, stacktrace)
> >>> >> >> NoSuchElementException: Message: no such element: Unable to
> locate
> >>> >> >> element: {"method":"xpath","selector":"//div[contains(@class,
> >>> >> >> "sql-editor-message") and contains(string(), "COMMIT")]"}
> >>> >> >> (Session info: chrome=58.0.3029.110)
> >>> >> >> (Driver info: chromedriver=2.29.461585
> >>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X
> 10.12.3
> >>> >> >> x86_64)
> >>> >> >>
> >>> >> > I have updated Auto rollback enabled test in this patch.
> >>> >> >
> >>> >> >>
> >>> >> >> Relevant screenshots attached.
> >>> >> >>
> >>> >> >> - Can you tidy up the regression output a little please? Instead
> of:
> >>> >> >>
> >>> >> >> -------
> >>> >> >> runTest
> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
> >>> >> >> Query tool feature test ... On demand result set on scrolling...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> On demand result set on grid select all...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> On demand result set on column select all...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Explain query...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Explain query with verbose...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Explain query with costs...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Explain analyze query...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Explain analyze query with buffers...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Explain analyze query with timing...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Auto commit disabled...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Auto commit enabled...
> >>> >> >> OK.
> >>> >> >>
> >>> >> >> Auto rollback enabled...
> >>> >> >> ERROR
> >>> >> >> -------
> >>> >> >>
> >>> >> >> Something like:
> >>> >> >>
> >>> >> >> -------
> >>> >> >> runTest
> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
> >>> >> >> Query tool feature test ...
> >>> >> >> On demand result set on scrolling... OK.
> >>> >> >> On demand result set on grid select all... OK.
> >>> >> >> On demand result set on column select all... OK.
> >>> >> >> Explain query... OK.
> >>> >> >> Explain query with verbose... OK.
> >>> >> >> Explain query with costs... OK.
> >>> >> >> Explain analyze query... OK.
> >>> >> >> Explain analyze query with buffers... OK.
> >>> >> >> Explain analyze query with timing... OK.
> >>> >> >> Auto commit disabled... OK.
> >>> >> >> Auto commit enabled... OK.
> >>> >> >> Auto rollback enabled... ERROR
> >>> >> >> --------
> >>> >> >>
> >>> >> > Fixed.
> >>> >> >
> >>> >> >
> >>> >> >>
> >>> >> >> Thanks!
> >>> >> >>
> >>> >> >> --
> >>> >> >> Dave Page
> >>> >> >> Blog: http://pgsnake.blogspot.com
> >>> >> >> Twitter: @pgsnake
> >>> >> >>
> >>> >> >> EnterpriseDB UK: http://www.enterprisedb.com
> >>> >> >> The Enterprise PostgreSQL Company
> >>> >> >
> >>> >> >
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Dave Page
> >>> >> Blog: http://pgsnake.blogspot.com
> >>> >> Twitter: @pgsnake
> >>> >>
> >>> >> EnterpriseDB UK: http://www.enterprisedb.com
> >>> >> The Enterprise PostgreSQL Company
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Dave Page
> >>> Blog: http://pgsnake.blogspot.com
> >>> Twitter: @pgsnake
> >>>
> >>> EnterpriseDB UK: http://www.enterprisedb.com
> >>> The Enterprise PostgreSQL Company
> >>
> >>
> >
> >
> >
> > --
> > Dave Page
> > Blog: http://pgsnake.blogspot.com
> > Twitter: @pgsnake
> >
> > EnterpriseDB UK: http://www.enterprisedb.com
> > The Enterprise PostgreSQL Company
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>