Re: Re: [pgAdmin4][Patch]: Fixes #1986 - Properly handle non-ascii characters while loading & saving file

Hi Dave,

The issues for RM#1986 & RM#1940 has been fixed and combined into a single
patch as they are related.

This patch is tested on Mac OSX, Ubuntu and Windows (py27 and py35).
Everything is working fine.
If there is still any issue. Please let me know.

​Please find updated patch and review.​

On Fri, Nov 25, 2016 at 3:23 PM, Dave Page <dpage(at)pgadmin(dot)org> wrote:

> Hi
>
> On Thu, Nov 24, 2016 at 10:11 AM, Surinder Kumar
> <surinder(dot)kumar(at)enterprisedb(dot)com> wrote:
> > Please ignore above patch. because it breaks when we save a file with
> name
> > containing ascii chars in py35
> > It is fixed. Please find attached patch and review.
> >
> > On Thu, Nov 24, 2016 at 2:56 PM, Surinder Kumar
> > <surinder(dot)kumar(at)enterprisedb(dot)com> wrote:
> >>
> >> Hi
> >>
> >> Issue:
> >> The content will not write to disk if it contains ascii characters.
> >>
> >> Solution:
> >> Encode/Decode the content before writing/reading the file.
> >>
> >> By the way, the file/folder name should also encode/decode, a similar
> >> issue is already logged
> >> #1940. I will work on it.
> >>
> >> Please find attached path and review.
>
> I'm running into all sorts of problems testing this. I'm on Mac, using
> Apple's Python 2.7.10 build.
>
> 1) Sometimes when opening the file manager to open a file, I get the
> following exception. It appears (but I'm not certain) to be when I've
> already opened a file with non-ASCII chars in the name and I've been
> prompted to discard the changes.
>
> 2016-11-25 09:42:20,087: INFO werkzeug: 127.0.0.1 - - [25/Nov/2016
> 09:42:20] "GET /file_manager/ HTTP/1.1" 500 -
> Traceback (most recent call last):
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 2000, in __call__
> return self.wsgi_app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1991, in wsgi_app
> response = self.make_response(self.handle_exception(e))
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1567, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1988, in wsgi_app
> response = self.full_dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1643, in full_dispatch_request
> response = self.process_response(response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1864, in process_response
> self.save_session(ctx.session, response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 926, in save_session
> return self.session_interface.save_session(self, session, response)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/session.py", line
> 267, in save_session
> self.manager.put(session)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/session.py", line
> 144, in put
> self.parent.put(session)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/utils/session.py", line
> 220, in put
> f
> RuntimeError: dictionary changed size during iteration
>
​I tried to reproduce with given steps. but unfortunately I couldn't
reproduce.​ If possible please provide more information.

>
> 2) If I use "Save As" and select an existing file with non-ASCII chars
> in the name, the filename box shows an encoded filename. See
> screenshot 1.
>
​Fixed.​

>
> 3) If I open an existing file with non-ASCII chars in the name, the
> title of the Query Tool tab is shown encoded. See screenshot 2. Note
> that if I save a new file, the name is displayed correctly.
>
​Fixed.​

>
> 4) Just selecting a file to open (without then clicking open) in the
> file manager gives the following exception:
>
> 2016-11-25 09:51:37,585: INFO werkzeug: 127.0.0.1 - - [25/Nov/2016
> 09:51:37] "POST /file_manager/filemanager/2265288/ HTTP/1.1" 500 -
> Traceback (most recent call last):
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 2000, in __call__
> return self.wsgi_app(environ, start_response)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1991, in wsgi_app
> response = self.make_response(self.handle_exception(e))
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1567, in handle_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1988, in wsgi_app
> response = self.full_dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1641, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1544, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1639, in full_dispatch_request
> rv = self.dispatch_request()
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask/app.py",
> line 1625, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
> ges/flask_login.py",
> line 792, in decorated_view
> return func(*args, **kwargs)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/misc/file_manager/__i
> nit__.py",
> line 820, in file_manager
> return getattr(myFilemanager, mode)(**kwargs)
> File "/Users/dpage/git/pgadmin4/web/pgadmin/misc/file_manager/__i
> nit__.py",
> line 518, in getinfo
> orig_path = "{0}{1}".format(self.dir, path)
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u0303' in
> position 18: ordinal not in range(128)
>
​Fixed.​

*RM#1940 issues reported:*
5) If I try to create a new folder with non-ASCII chars it the name, I get:

2016-11-25 10:42:55,074: INFO werkzeug: 127.0.0.1 - - [25/Nov/2016
10:42:55] "GET /file_manager/filemanager/7000827/?mode=addfolder&path=/&nam
e=u%CC%88brig_wei%C3%9F
HTTP/1.1" 500 -
Traceback (most recent call last):
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 2000, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1991, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1567, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask_login.py",
line 792, in decorated_view
return func(*args, **kwargs)
File "/Users/dpage/git/pgadmin4/web/pgadmin/misc/file_manager/__i
nit__.py",
line 842, in file_manager
return getattr(myFilemanager, mode)(**kwargs)
File "/Users/dpage/git/pgadmin4/web/pgadmin/misc/file_manager/__i
nit__.py",
line 756, in addfolder
newPath = dir + '/' + path + newName + '/'
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position
1: ordinal not in range(128)
Fixed.

6) If I try to open a folder (by double clicking it) with non-ASCII
chars in the name, the location bar is updated but the file/folder
list is not redrawn. I get the following exception:

2016-11-25 10:46:15,651: INFO werkzeug: 127.0.0.1 - - [25/Nov/2016
10:46:15] "POST /file_manager/filemanager/7000827/ HTTP/1.1" 500 -
Traceback (most recent call last):
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 2000, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1991, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1567, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask/app.py",
line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
ges/flask_login.py",
line 792, in decorated_view
return func(*args, **kwargs)
File "/Users/dpage/git/pgadmin4/web/pgadmin/misc/file_manager/__i
nit__.py",
line 842, in file_manager
return getattr(myFilemanager, mode)(**kwargs)
File "/Users/dpage/git/pgadmin4/web/pgadmin/misc/file_manager/__i
nit__.py",
line 561, in getfolder
filelist = self.list_filesystem(dir, path, trans_data, file_type)
File "/Users/dpage/git/pgadmin4/web/pgadmin/misc/file_manager/__i
nit__.py",
line 433, in list_filesystem
orig_path = "{0}{1}".format(dir, path)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
1-6: ordinal not in range(128)
Fixed.

>
> Please test thoroughly in both Python 2 and 3 before re-submitting.
>
> Thanks.
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>