Re: [pgadmin-hackers] Re: Server side cursor limitations for on demand loading of data in query tool [RM2137] [pgAdmin4]

I'm sorry, it needs rebasing again. If you can do it quickly, I'll
make sure it's the next patch I work on in that area.

Thanks.

On Mon, Jun 26, 2017 at 5:16 AM, Harshal Dhumal
<harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
> Hi Dave,
>
> Please find updated rebased patch for RM2137
>
> On Fri, Jun 23, 2017 at 9:00 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>>
>> Hi Harshal,
>>
>> When can we expect an updated version of this patch? I think it's
>> important to get this into the next release.
>>
>> Thanks!
>>
>> On Fri, Jun 16, 2017 at 10:55 AM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>> > Hi,
>> >
>> > That's better - the failures are far less random now :-). I got the
>> > following two though, on both PG and EPAS 9.5:
>> >
>> > ======================================================================
>> > ERROR: runTest
>> > (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> > Query tool feature test
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> > File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> > line 95, in runTest
>> > self._query_tool_explain_analyze_buffers()
>> > File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> > line 443, in _query_tool_explain_analyze_buffers
>> > canvas.find_element_by_xpath("//*[contains(string(), 'Shared Read
>> > Blocks')]")
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 260, in find_element_by_xpath
>> > return self.find_element(by=By.XPATH, value=xpath)
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 508, in find_element
>> > {"using": by, "value": value})['value']
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 491, in _execute
>> > return self._parent.execute(command, params)
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> > line 238, in execute
>> > self.error_handler.check_response(response)
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> > line 193, in check_response
>> > raise exception_class(message, screen, stacktrace)
>> > NoSuchElementException: Message: no such element: Unable to locate
>> > element: {"method":"xpath","selector":"//*[contains(string(), 'Shared
>> > Read Blocks')]"}
>> > (Session info: chrome=58.0.3029.110)
>> > (Driver info: chromedriver=2.29.461585
>> > (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> > x86_64)
>> >
>> >
>> > ======================================================================
>> > ERROR: runTest
>> > (pgadmin.feature_tests.view_data_dml_queries.CheckForViewDataTest)
>> > Validate Insert, Update operations in View data with given test data
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> > File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 104, in runTest
>> > self._add_row()
>> > File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 255, in _add_row
>> > self._update_cell(cell_xpath, config_data[str(idx)])
>> > File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 164, in _update_cell
>> > cell_el = self.page.find_by_xpath(xpath)
>> > File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 122, in find_by_xpath
>> > return self.wait_for_element(lambda driver:
>> > driver.find_element_by_xpath(xpath))
>> > File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 205, in wait_for_element
>> > return self._wait_for("element to exist", element_if_it_exists)
>> > File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 255, in _wait_for
>> > "Timed out waiting for " + waiting_for_message)
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/support/wait.py",
>> > line 71, in until
>> > value = method(self._driver)
>> > File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 200, in element_if_it_exists
>> > if element.is_displayed() and element.is_enabled():
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 358, in is_displayed
>> > return self._execute(Command.IS_ELEMENT_DISPLAYED)['value']
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 491, in _execute
>> > return self._parent.execute(command, params)
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> > line 238, in execute
>> > self.error_handler.check_response(response)
>> > File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> > line 193, in check_response
>> > raise exception_class(message, screen, stacktrace)
>> > StaleElementReferenceException: Message: stale element reference:
>> > element is not attached to the page document
>> > (Session info: chrome=58.0.3029.110)
>> > (Driver info: chromedriver=2.29.461585
>> > (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> > x86_64)
>> >
>
>
> I checked my feature test cases for any database server version/type (PG,
> EPAS) specific failures and I found that test cases are working fine on all
> of them. However I have slightly modified test case to overcome above
> mentioned failures.
>
>>
>> >
>> > On 9.4 everything passes. On 9.6, I get one failure:
>> >
>> > ======================================================================
>> > FAIL: runTest
>> > (pgadmin.feature_tests.view_data_dml_queries.CheckForViewDataTest)
>> > Validate Insert, Update operations in View data with given test data
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> > File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 105, in runTest
>> > self._verify_row_data(True)
>> > File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 282, in _verify_row_data
>> > self.assertEquals(cells[idx], config_data[str(idx)][1])
>> > AssertionError: u'[default]' != u'1'
>> > - [default]
>> > + 1
>> >
>
> I guess Khushboo has fixed this recently.
>
>
>>
>> > Screenshots attached.
>> >
>> > On Fri, Jun 16, 2017 at 7:54 AM, Harshal Dhumal
>> > <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> >> Hi Dave,
>> >>
>> >> Please find attached patch where I have added timeout of 2 seconds
>> >> before
>> >> selecting object menu in connect to server function.
>> >>
>> >> Note: Apply this patch on top of previous patch for on demand loading
>> >> feature.
>> >>
>> >>
>> >> --
>> >> Harshal Dhumal
>> >> Sr. Software Engineer
>> >>
>> >> EnterpriseDB India: http://www.enterprisedb.com
>> >> The Enterprise PostgreSQL Company
>> >>
>> >> On Fri, Jun 16, 2017 at 2:25 AM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>> >>>
>> >>> Sounds good, thanks.
>> >>>
>> >>> On Thu, Jun 15, 2017 at 9:54 PM, Harshal Dhumal
>> >>> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> >>> > Hi
>> >>> >
>> >>> > On Fri, Jun 16, 2017 at 2:07 AM, Dave Page <dpage(at)pgadmin(dot)org>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi
>> >>> >>
>> >>> >> On Thu, Jun 15, 2017 at 9:30 PM, Harshal Dhumal
>> >>> >> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> >>> >> > Hi Dave,
>> >>> >> >
>> >>> >> > Please find attached updated patch.
>> >>> >> >
>> >>> >> > On Thu, Jun 15, 2017 at 3:58 PM, Dave Page <dpage(at)pgadmin(dot)org>
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> Hi
>> >>> >> >>
>> >>> >> >> On Wed, Jun 14, 2017 at 11:36 PM, Harshal Dhumal
>> >>> >> >> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> >>> >> >>>
>> >>> >> >>> Hi Dave,
>> >>> >> >>>
>> >>> >> >>> Please find rebased patch for RM2137.
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Looking very good. The only issues I see are:
>> >>> >> >>
>> >>> >> >> - The row headers should auto-size such that they can display
>> >>> >> >> the
>> >>> >> >> row
>> >>> >> >> numbers if the last row was displayed. E.g. if there are 12345
>> >>> >> >> rows
>> >>> >> >> in
>> >>> >> >> total, then the row header should be sized to display 5 digits.
>> >>> >> >>
>> >>> >> > Fixed.
>> >>> >> >
>> >>> >> >
>> >>> >> >>
>> >>> >> >> - The tests are comprehensive, which is awesome. However, every
>> >>> >> >> time
>> >>> >> >> I
>> >>> >> >> ran
>> >>> >> >> them, at least one of the feature tests failed. Unfortunately,
>> >>> >> >> it
>> >>> >> >> was a
>> >>> >> >> different one each time. In the last two runs, I got:
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ======================================================================
>> >>> >> >> ERROR: runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ----------------------------------------------------------------------
>> >>> >> >> Traceback (most recent call last):
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/regression/feature_utils/base_feature_test.py",
>> >>> >> >> line 40, in setUp
>> >>> >> >> self.before()
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 40, in before
>> >>> >> >> self._connects_to_server()
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 144, in _connects_to_server
>> >>> >> >> self.page.driver.find_element_by_link_text("Create"))\
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 319, in find_element_by_link_text
>> >>> >> >> return self.find_element(by=By.LINK_TEXT, value=link_text)
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 756, in find_element
>> >>> >> >> 'value': value})['value']
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 238, in execute
>> >>> >> >> self.error_handler.check_response(response)
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >>> >> >> line 193, in check_response
>> >>> >> >> raise exception_class(message, screen, stacktrace)
>> >>> >> >> NoSuchElementException: Message: no such element: Unable to
>> >>> >> >> locate
>> >>> >> >> element: {"method":"link text","selector":"Create"}
>> >>> >> >> (Session info: chrome=58.0.3029.110)
>> >>> >> >> (Driver info: chromedriver=2.29.461585
>> >>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X
>> >>> >> >> 10.12.3
>> >>> >> >> x86_64)
>> >>> >> >>
>> >>> >> > I checked code from _connects_to_server function which is common
>> >>> >> > in
>> >>> >> > all
>> >>> >> > features test cases. I didn't find anything wrong with this. If
>> >>> >> > there
>> >>> >> > is
>> >>> >> > a
>> >>> >> > bug in this function then all feature test must fail.
>> >>> >> > Let me know if you are getting failure consistently in
>> >>> >> > _connects_to_server
>> >>> >> > function.
>> >>> >>
>> >>> >> I wondered if that one is a race condition. Do we need a short
>> >>> >> delay
>> >>> >> before clicking the Object menu? I have seen this occasionally
>> >>> >> before.
>> >>> >
>> >>> >
>> >>> > OK. In that case let's try putting 1-2 second delay and observer
>> >>> > behaviour.
>> >>> > I'll send separate patch for this tomorrow as this is not related to
>> >>> > on
>> >>> > demand query result feature or its test cases.
>> >>> >
>> >>> >>
>> >>> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ======================================================================
>> >>> >> >> ERROR: runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ----------------------------------------------------------------------
>> >>> >> >> Traceback (most recent call last):
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 119, in runTest
>> >>> >> >> self._query_tool_auto_rollback_enabled()
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 697, in _query_tool_auto_rollback_enabled
>> >>> >> >> '//div[contains(@class, "sql-editor-message") and
>> >>> >> >> contains(string(),
>> >>> >> >> "COMMIT")]'
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 295, in find_element_by_xpath
>> >>> >> >> return self.find_element(by=By.XPATH, value=xpath)
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 756, in find_element
>> >>> >> >> 'value': value})['value']
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 238, in execute
>> >>> >> >> self.error_handler.check_response(response)
>> >>> >> >> File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >>> >> >> line 193, in check_response
>> >>> >> >> raise exception_class(message, screen, stacktrace)
>> >>> >> >> NoSuchElementException: Message: no such element: Unable to
>> >>> >> >> locate
>> >>> >> >> element: {"method":"xpath","selector":"//div[contains(@class,
>> >>> >> >> "sql-editor-message") and contains(string(), "COMMIT")]"}
>> >>> >> >> (Session info: chrome=58.0.3029.110)
>> >>> >> >> (Driver info: chromedriver=2.29.461585
>> >>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X
>> >>> >> >> 10.12.3
>> >>> >> >> x86_64)
>> >>> >> >>
>> >>> >> > I have updated Auto rollback enabled test in this patch.
>> >>> >> >
>> >>> >> >>
>> >>> >> >> Relevant screenshots attached.
>> >>> >> >>
>> >>> >> >> - Can you tidy up the regression output a little please? Instead
>> >>> >> >> of:
>> >>> >> >>
>> >>> >> >> -------
>> >>> >> >> runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test ... On demand result set on scrolling...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> On demand result set on grid select all...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> On demand result set on column select all...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query with verbose...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query with costs...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query with buffers...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query with timing...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto commit disabled...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto commit enabled...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto rollback enabled...
>> >>> >> >> ERROR
>> >>> >> >> -------
>> >>> >> >>
>> >>> >> >> Something like:
>> >>> >> >>
>> >>> >> >> -------
>> >>> >> >> runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test ...
>> >>> >> >> On demand result set on scrolling... OK.
>> >>> >> >> On demand result set on grid select all... OK.
>> >>> >> >> On demand result set on column select all... OK.
>> >>> >> >> Explain query... OK.
>> >>> >> >> Explain query with verbose... OK.
>> >>> >> >> Explain query with costs... OK.
>> >>> >> >> Explain analyze query... OK.
>> >>> >> >> Explain analyze query with buffers... OK.
>> >>> >> >> Explain analyze query with timing... OK.
>> >>> >> >> Auto commit disabled... OK.
>> >>> >> >> Auto commit enabled... OK.
>> >>> >> >> Auto rollback enabled... ERROR
>> >>> >> >> --------
>> >>> >> >>
>> >>> >> > Fixed.
>> >>> >> >
>> >>> >> >
>> >>> >> >>
>> >>> >> >> Thanks!
>> >>> >> >>
>> >>> >> >> --
>> >>> >> >> Dave Page
>> >>> >> >> Blog: http://pgsnake.blogspot.com
>> >>> >> >> Twitter: @pgsnake
>> >>> >> >>
>> >>> >> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> >> >> The Enterprise PostgreSQL Company
>> >>> >> >
>> >>> >> >
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Dave Page
>> >>> >> Blog: http://pgsnake.blogspot.com
>> >>> >> Twitter: @pgsnake
>> >>> >>
>> >>> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> >> The Enterprise PostgreSQL Company
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Dave Page
>> >>> Blog: http://pgsnake.blogspot.com
>> >>> Twitter: @pgsnake
>> >>>
>> >>> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> The Enterprise PostgreSQL Company
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Dave Page
>> > Blog: http://pgsnake.blogspot.com
>> > Twitter: @pgsnake
>> >
>> > EnterpriseDB UK: http://www.enterprisedb.com
>> > The Enterprise PostgreSQL Company
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company