Re: Question on inserting non-ascii strings

On Thu, May 14, 2009 at 07:39:41PM -0400, Steven Lembark wrote:
>
> > Which "warnings" are you talking about? I thought you said you had
> > plain SQL working OK, but were struggling to pass parameters containing
> > UTF-8 encoded characters.
> >
> > Are you sure that your Perl code is passing the string encoded as UTF8?
>
> Excellent point: Perl will only pass through the
> converted UTF8, if I remember to convert it from
> unicode!

Probably a minor point; but Unicode doesn't mean much to me here,
Unicode defines a set of code-points and their mapping to "characters".
These code-points are always encoded in some form whether that be UTF-8,
UTF-16 or something else. Strings may be handled in Perl as Unicode
strings of unspecified encoding (I don't know) and hence you need to
specify what encoding you want them in if you ask for an array of bytes.

> The earlier version of the database used SQL_ASCII
> and took the unicode byte as-is with a nastygram
> about trans-ascii byte -- which led me down the
> path of E'xxx'. While trying to make that work I'd
> accidentally fat-figered out the utf8 conversion
> attempting to comment it out.

Doh, easy to do. One things that I'd be tempted to do to make sure your
code is doing the right thing would be to test that the characters are
correct using another client--psql would be a reasonable choice. That
way you can have some more confidence that if you have to get other code
to talk to the database it's actually doing what you want it to be doing
and it's not just Perl being consistent with itself, as it should be.

I know I've saved myself a few times doing similar things like this.

--
Sam http://samason.me.uk/