Re: Patch from RM1983 [pgAdmin4]

Hi,

Please find updated patch for unicode issue on python 2.7

I have tested with below scenarios for all nodes (except database and
Login/Group Role as these are stored in shared catalogs as for now we are
not considering encoding issues for these two nodes)

scenarios: Create and Update nodes

Note: Apply patches which I have just sent (with subject pgAdmin4 minor
issues [pgAdmin4]) before applying this patch.

--
*Harshal Dhumal*
*Software Engineer*

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Feb 17, 2017 at 4:22 PM, Murtuza Zabuawala <
murtuza(dot)zabuawala(at)enterprisedb(dot)com> wrote:

> Hi Harshal,
>
> *Python3.5*
> 1) I installed PG_9.4 with locale: Arabic, Saudi Arabia,
> - When I try to create new database with name "السلام" -> with 'template
> 0' as template , it fails to open, see screenshot
> - Backup/Restore fails, refer screenshot
>
> I tested INSET/UPDATE/DELETE and other common operations and seems to be
> working.
> Attaching screenshots.
>
>
> 2) I installed PG_9.5 with locale: Chinese, China
> - I am not able to create new DB with Chinese name same as above mentioned
> Same as (1), most parts are working with patch
> Attaching screenshots.
>
> 3) I installed PG_9.5 with locale: Japanese, Japan
> - I am not able to create new DB with Japanese name same as above mentioned
> Same as (1), most parts are working with patch
> Attaching screenshots.
>
>
> *Python2.7 w**ith 3 server mentioned above*
> Works fine most the places but also fails at some places with Unicode
> errors, As I observed it fails when we receive the data from DB server (by
> running properties.sql) and we try to render the data in template as we
> received from DB server.
> Screenshots attached.
>
> *Common Exceptions:*
>
> {{conn|qtIdent(c.name)}} {{ GET_TYPE.CREATE_TYPE_SQL(conn, c.cltype,
> c.attlen, c.attprecision, c.hasSqrBracket) }}{% if c.collspcname %} COLLATE
> {{c.collspcname}}{% endif %}{% if c.attnotnull %} NOT NULL{% endif %}{% if
> c.defval %} DEFAULT {{c.defval}}{% endif %}
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 10:
> ordinal not in range(128)
> Traceback (most recent call last):
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
> line 2000, in __call__
> return self.wsgi_app(environ, start_response)
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
> line 1991, in wsgi_app
> response = self.make_response(self.handle_exception(e))
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
> line 1567, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
> line 1988, in wsgi_app
> response = self.full_dispatch_request()
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
> line 1641, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
> line 1544, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
> line 1639, in full_dispatch_request
> rv = self.dispatch_request()
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
> line 1625, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/views.py",
> line 84, in view
> return self.dispatch_request(*args, **kwargs)
> File "/home/murtuza/projects/pgadmin4/web/pgadmin/browser/utils.py",
> line 235, in dispatch_request
> return method(*args, **kwargs)
> File "/home/murtuza/projects/pgadmin4/web/pgadmin/browser/
> server_groups/servers/databases/schemas/functions/__init__.py", line 341,
> in wrap
> return f(*args, **kwargs)
> File "/home/murtuza/projects/pgadmin4/web/pgadmin/browser/
> server_groups/servers/databases/schemas/functions/__init__.py", line
> 1021, in sql
> query_for="sql_panel")
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/templating.py",
> line 134, in render_template
> context, ctx.app)
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/templating.py",
> line 116, in _render
> rv = template.render(context)
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/jinja2/environment.py",
> line 969, in render
> return self.environment.handle_exception(exc_info, True)
> File "/home/murtuza/projects/venv2/lib/python2.7/site-packages/jinja2/environment.py",
> line 742, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/home/murtuza/projects/pgadmin4/web/pgadmin/browser/
> server_groups/servers/databases/schemas/functions/
> templates/function/pg/sql/9.5_plus/create.sql", line 7, in top-level
> template code
> CREATE{% if query_type is defined %}{{' OR REPLACE'}}{% endif %}
> FUNCTION {{func_def}}
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0:
> ordinal not in range(128)
>
>
> --
> Regards,
> Murtuza Zabuawala
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
> On Thu, Feb 9, 2017 at 4:34 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>
>> Harshal,
>>
>> I had a brief discussion with Ashesh about this. He's going to setup a
>> call next week for us to discuss a plan of action for eliminating any
>> remaining encoding issues including this one. I'm putting this on hold
>> until then.
>>
>> Thanks.
>>
>> On Mon, Feb 6, 2017 at 2:54 PM, Harshal Dhumal
>> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> > Hi Dave,
>> >
>> > --
>> > Harshal Dhumal
>> > Software Engineer
>> >
>> > EnterpriseDB India: http://www.enterprisedb.com
>> > The Enterprise PostgreSQL Company
>> >
>> > On Mon, Feb 6, 2017 at 6:48 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
>> >>
>> >> Hi
>> >>
>> >> On Mon, Feb 6, 2017 at 12:57 PM, Harshal Dhumal
>> >> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
>> >> > Hi,
>> >> >
>> >> > Please find attached patch for RM 1983.
>> >> >
>> >> > This issue only occurs when database encoding is other than utf-8
>> >> >
>> >> > Also other issue was when we use connection of database with encoding
>> >> > other
>> >> > than utf-8 to retrieve data from cluster table/s which has encoding
>> >> > utf-8
>> >> > (e.g. pg_database) then data was not decoded properly.
>> >>
>> >> The code makes an assumption that pg_database is always utf-8 encoded.
>> >> I don't believe that is correct - I believe it's the encoding used in
>> >> the database from which the new database was created. The general
>> >> advice is that users should avoid using non-7bit ASCII characters in
>> >> shared catalogs, e.g. databases and comments etc.
>> >
>> >
>> > Ok.
>> >
>> > Let me split this into two issues:
>> > i) RM1983 for which I have attached updated patch. (basically I removed
>> > changes related to decode data retried from pg_database when connection
>> > encoding is other than utf-8)
>> > ii) Support to allow user to use non-&bit ASCII characters in shared
>> > catalogs with the help of pgAdmin4.
>> >
>> > Regarding your statement about pg_database "I believe it's the encoding
>> used
>> > in the database from which the new database was created.". I found it
>> > little-bit confusing for me (correct me if i'm wrong); As mentioned here
>> > there is only one copy of pg_database per cluster. So I assume
>> pg_database
>> > is created when we initialize database cluster and not when we create
>> new
>> > database.
>> >
>> >
>> >>
>> >>
>> >> See
>> >> https://www.postgresql.org/message-id/flat/20160216163833.
>> GF31273%40momjian(dot)us#20160216163833(dot)GF31273(at)momjian(dot)us
>> >> for more info for example.
>> >>
>> >> Did pgAdmin 3 just assume it was UTF-8? I suspect it did - and that
>> >> just happened to work in most cases.
>> >>
>> >> --
>> >> Dave Page
>> >> Blog: http://pgsnake.blogspot.com
>> >> Twitter: @pgsnake
>> >>
>> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >> The Enterprise PostgreSQL Company
>> >
>> >
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>>
>> --
>> Sent via pgadmin-hackers mailing list (pgadmin-hackers(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgadmin-hackers
>>
>
>