Re: [pgAdmin4][RM#3289] Can't query SQL_ASCII database.

Hi Dave,

Attached is the updated patch. (Playing with encodings is not at all fun :(
)

On Tue, Jun 19, 2018 at 2:23 AM, Dave Page <dpage(at)pgadmin(dot)org> wrote:

> Hi
>
> On Mon, Jun 18, 2018 at 2:14 PM, Aditya Toshniwal <aditya.toshniwal@
> enterprisedb.com> wrote:
>
>> Hi Hackers,
>>
>> Attached is the updated patch which includes the fix for Download CSV
>> fail in SQL_ASCII database, which is RM3250
>> <https://redmine.postgresql.org/issues/3250>
>> This should fix RM3289 and RM3250. As they interrelated, sending the
>> combined patch.
>> Kindly review.
>>
>
> This is definitely looking better - both view and save now work as
> expected. However, using the test data the I posted upthread, if I try to
> edit a value (in this case by adding a couple of chars to the end of the
> data in row 2) I get:
>
It should fix the error.

>
> 2018-06-18 16:41:40,895: SQL pgadmin: Execute (void) for server #1 -
> DB:ascii (Query-id: 3093186):
> UPDATE public.ascii SET
> data = %(data)s::text WHERE
> id = '2';
> 2018-06-18 16:41:41,027: INFO werkzeug: 127.0.0.1 - - [18/Jun/2018
> 16:41:41] "POST /sqleditor/save/2805058 HTTP/1.1" 500 -
> 2018-06-18 16:41:41,042: ERROR werkzeug: Error on request:
> Traceback (most recent call last):
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 270, in run_wsgi
> execute(self.server.app)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
> line 258, in execute
> application_iter = app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1997, in __call__
> return self.wsgi_app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1985, in wsgi_app
> response = self.handle_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1540, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1982, in wsgi_app
> response = self.full_dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1614, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1517, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1612, in full_dispatch_request
> rv = self.dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask/app.py",
> line 1598, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/flask_login.py",
> line 792, in decorated_view
> return func(*args, **kwargs)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py",
> line 776, in save
> default_conn)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/tools/sqleditor/command.py",
> line 829, in save
> item['sql'], item['data'])
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
> line 975, in execute_void
> self.__internal_blocking_execute(cur, query, params)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
> line 629, in __internal_blocking_execute
> cur.execute(query, params)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/cursor.py",
> line 176, in execute
> return _cursor.execute(self, query, params)
> UnicodeEncodeError: 'ascii' codec can't encode characters in position
> 19-21: ordinal not in range(128)
>
>
>>
>> On Fri, Jun 15, 2018 at 2:33 PM, Aditya Toshniwal <
>> aditya(dot)toshniwal(at)enterprisedb(dot)com> wrote:
>>
>>> Hi Dave,
>>>
>>> On Fri, Jun 15, 2018 at 2:08 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>>>
>>>> Hi
>>>>
>>>> On Thu, Jun 14, 2018 at 7:05 AM, Aditya Toshniwal <
>>>> aditya(dot)toshniwal(at)enterprisedb(dot)com> wrote:
>>>>
>>>>> I am sorry I missed the attachment. :(
>>>>> PFA.
>>>>>
>>>>
>>>> It looks like the encoding names are getting munged somewhere. I see
>>>> you've accounted for that to some degree in connection.py (you have both
>>>> SQL_ASCII/MULE_INTERNAL and SQLASCII/MULEINTERNAL), however it doesn't seem
>>>> to be enough as I'm getting the following error when trying to download CSV
>>>> from the query tool. Can we ensure that conn.encoding contains an un-munged
>>>> value at all times, or is that coming from psycopg2?
>>>>
>>> ​That is done by pyscopg2 and conn.encoding is a psycopg2 connection
>>> property.​
>>>
>>>
>>>>
>>>> 2018-06-15 09:32:28,799: INFO werkzeug: 127.0.0.1 - - [15/Jun/2018
>>>> 09:32:28] "GET /sqleditor/query_tool/download
>>>> /2732923?query=SELECT%20*%20FROM%20public.sql_ascii%0AORDER%
>>>> 20BY%20id%20ASC%20&filename=sql_ascii.csv HTTP/1.1" 500 -
>>>> 2018-06-15 09:32:28,801: ERROR werkzeug: Error on request:
>>>> Traceback (most recent call last):
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>> line 270, in run_wsgi
>>>> execute(self.server.app)
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/serving.py",
>>>> line 260, in execute
>>>> for data in application_iter:
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wsgi.py",
>>>> line 870, in __next__
>>>> return self._next()
>>>> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/werkzeug/wrappers.py",
>>>> line 82, in _iter_encoded
>>>> for item in iterable:
>>>> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/driver/psycopg2/connection.py",
>>>> line 752, in gen
>>>> column_name = column_name.decode(conn_encoding)
>>>> LookupError: unknown encoding: SQLASCII
>>>>
>>>
>>> ​This is because there is code bug here. Below is code used to decode a
>>> column name. Connection encoding and python encoding are two different
>>> things. Python does not know what SQLASCII is. This will work with UTF-8
>>> because python has decoder with same name. I tried to download CSV with the
>>> original code without changes and it fails there too. I will fix this and
>>> will send the updated patch. I should have checked this.
>>> conn_encoding = cur.connection.encoding
>>> column_name = column_name.decode(conn_encoding)​
>>>
>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Aditya Toshniwal
>>> Software Engineer | EnterpriseDB Software Solutions | Pune
>>> "Don't Complain about Heat, Plant a tree"
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Aditya Toshniwal
>> Software Engineer | EnterpriseDB Software Solutions | Pune
>> "Don't Complain about Heat, Plant a tree"
>>
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

--
Thanks and Regards,
Aditya Toshniwal
Software Engineer | EnterpriseDB Software Solutions | Pune
"Don't Complain about Heat, Plant a tree"