Re: Patch from RM1983 [pgAdmin4]

Hi Harshal,

*Python3.5*
1) I installed PG_9.4 with locale: Arabic, Saudi Arabia,
- When I try to create new database with name "السلام" -> with 'template 0'
as template , it fails to open, see screenshot
- Backup/Restore fails, refer screenshot

I tested INSET/UPDATE/DELETE and other common operations and seems to be
working.
Attaching screenshots.

2) I installed PG_9.5 with locale: Chinese, China
- I am not able to create new DB with Chinese name same as above mentioned
Same as (1), most parts are working with patch
Attaching screenshots.

3) I installed PG_9.5 with locale: Japanese, Japan
- I am not able to create new DB with Japanese name same as above mentioned
Same as (1), most parts are working with patch
Attaching screenshots.

*Python2.7 w**ith 3 server mentioned above*
Works fine most the places but also fails at some places with Unicode
errors, As I observed it fails when we receive the data from DB server (by
running properties.sql) and we try to render the data in template as we
received from DB server.
Screenshots attached.

*Common Exceptions:*

{{conn|qtIdent(c.name)}} {{ GET_TYPE.CREATE_TYPE_SQL(conn, c.cltype,
c.attlen, c.attprecision, c.hasSqrBracket) }}{% if c.collspcname %} COLLATE
{{c.collspcname}}{% endif %}{% if c.attnotnull %} NOT NULL{% endif %}{% if
c.defval %} DEFAULT {{c.defval}}{% endif %}
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 10:
ordinal not in range(128)
Traceback (most recent call last):
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
line 2000, in __call__
return self.wsgi_app(environ, start_response)
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
line 1991, in wsgi_app
response = self.make_response(self.handle_exception(e))
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
line 1567, in handle_exception
reraise(exc_type, exc_value, tb)
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
line 1988, in wsgi_app
response = self.full_dispatch_request()
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
line 1639, in full_dispatch_request
rv = self.dispatch_request()
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/app.py",
line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/views.py",
line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/home/murtuza/projects/pgadmin4/web/pgadmin/browser/utils.py", line
235, in dispatch_request
return method(*args, **kwargs)
File
"/home/murtuza/projects/pgadmin4/web/pgadmin/browser/server_groups/servers/databases/schemas/functions/__init__.py",
line 341, in wrap
return f(*args, **kwargs)
File
"/home/murtuza/projects/pgadmin4/web/pgadmin/browser/server_groups/servers/databases/schemas/functions/__init__.py",
line 1021, in sql
query_for="sql_panel")
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/templating.py",
line 134, in render_template
context, ctx.app)
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/flask/templating.py",
line 116, in _render
rv = template.render(context)
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/jinja2/environment.py",
line 969, in render
return self.environment.handle_exception(exc_info, True)
File
"/home/murtuza/projects/venv2/lib/python2.7/site-packages/jinja2/environment.py",
line 742, in handle_exception
reraise(exc_type, exc_value, tb)
File
"/home/murtuza/projects/pgadmin4/web/pgadmin/browser/server_groups/servers/databases/schemas/functions/templates/function/pg/sql/9.5_plus/create.sql",
line 7, in top-level template code
CREATE{% if query_type is defined %}{{' OR REPLACE'}}{% endif %}
FUNCTION {{func_def}}
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0:
ordinal not in range(128)

--
Regards,
Murtuza Zabuawala
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Feb 9, 2017 at 4:34 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:

> Harshal,
>
> I had a brief discussion with Ashesh about this. He's going to setup a
> call next week for us to discuss a plan of action for eliminating any
> remaining encoding issues including this one. I'm putting this on hold
> until then.
>
> Thanks.
>
> On Mon, Feb 6, 2017 at 2:54 PM, Harshal Dhumal
> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
> > Hi Dave,
> >
> > --
> > Harshal Dhumal
> > Software Engineer
> >
> > EnterpriseDB India: http://www.enterprisedb.com
> > The Enterprise PostgreSQL Company
> >
> > On Mon, Feb 6, 2017 at 6:48 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:
> >>
> >> Hi
> >>
> >> On Mon, Feb 6, 2017 at 12:57 PM, Harshal Dhumal
> >> <harshal(dot)dhumal(at)enterprisedb(dot)com> wrote:
> >> > Hi,
> >> >
> >> > Please find attached patch for RM 1983.
> >> >
> >> > This issue only occurs when database encoding is other than utf-8
> >> >
> >> > Also other issue was when we use connection of database with encoding
> >> > other
> >> > than utf-8 to retrieve data from cluster table/s which has encoding
> >> > utf-8
> >> > (e.g. pg_database) then data was not decoded properly.
> >>
> >> The code makes an assumption that pg_database is always utf-8 encoded.
> >> I don't believe that is correct - I believe it's the encoding used in
> >> the database from which the new database was created. The general
> >> advice is that users should avoid using non-7bit ASCII characters in
> >> shared catalogs, e.g. databases and comments etc.
> >
> >
> > Ok.
> >
> > Let me split this into two issues:
> > i) RM1983 for which I have attached updated patch. (basically I removed
> > changes related to decode data retried from pg_database when connection
> > encoding is other than utf-8)
> > ii) Support to allow user to use non-&bit ASCII characters in shared
> > catalogs with the help of pgAdmin4.
> >
> > Regarding your statement about pg_database "I believe it's the encoding
> used
> > in the database from which the new database was created.". I found it
> > little-bit confusing for me (correct me if i'm wrong); As mentioned here
> > there is only one copy of pg_database per cluster. So I assume
> pg_database
> > is created when we initialize database cluster and not when we create new
> > database.
> >
> >
> >>
> >>
> >> See
> >> https://www.postgresql.org/message-id/flat/20160216163833.GF31273%
> 40momjian(dot)us#20160216163833(dot)GF31273(at)momjian(dot)us
> >> for more info for example.
> >>
> >> Did pgAdmin 3 just assume it was UTF-8? I suspect it did - and that
> >> just happened to work in most cases.
> >>
> >> --
> >> Dave Page
> >> Blog: http://pgsnake.blogspot.com
> >> Twitter: @pgsnake
> >>
> >> EnterpriseDB UK: http://www.enterprisedb.com
> >> The Enterprise PostgreSQL Company
> >
> >
>
>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
>
> --
> Sent via pgadmin-hackers mailing list (pgadmin-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgadmin-hackers
>