Re: Postgres, apps, special characters and UTF-8 encoding

On 03/07/2017 03:20 PM, Ken Tanzer wrote:
> Hi. I've got a recurring problem with character encoding for a
> Postgres-based web PHP app, and am hoping someone can clue me in or at
> least point me in the right direction. I'll confess upfront my
> understanding of encoding issues is extremely limited. Here goes.
>
> The app uses a Postgres database, UTF-8 encoded. Through their
> browsers, users can add and edit records often including text. Most of
> the time this works fine. Though sometimes this will fail with Postgres
> complaining, for example, "Could query with ... , The error text was:
> ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x20 0x67"
>
> So this generally happens when people copy and paste things out of their
> word documents and such.
>
> As I understand it, those are likely encoded in something non-UTF-8,
> like WIN-1251 or something. And that one way or another, the encoding
> needs to be translated before it can be placed into the database. I'm
> not clear how this is supposed to happen though. Automatically by the
> browser? Done in the app? Some other way? And if in the app, how is
> one supposed to know what the incoming encoding is?

I don't use PHP, but found this:

http://www.php.net/manual/en/function.mb-detect-encoding.php

and this:

http://php.net/manual/en/function.mb-convert-encoding.php

>
> Thanks in advance for any help or pointers.
>
> Ken
>
>
> --
> AGENCY Software
> A Free Software data system
> By and for non-profits
> /http://agency-software.org//
> /https://agency-software.org/demo/client/
> ken(dot)tanzer(at)agency-software(dot)org <mailto:ken(dot)tanzer(at)agency-software(dot)org>
> (253) 245-3801
>
> Subscribe to the mailing list
> <mailto:agency-general-request(at)lists(dot)sourceforge(dot)net?body=subscribe> to
> learn more about AGENCY or
> follow the discussion.

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com