Re: Invalid byte sequence for encoding "UTF8": 0xedbebf

-----Original Message-----
From: Albe Laurenz [mailto:laurenz(dot)albe(at)wien(dot)gv(dot)at]
Sent: Thursday, June 16, 2011 5:16 AM
To: BRUSSER Michael; pgsql-general(at)postgresql(dot)org
Subject: RE: [GENERAL] Invalid byte sequence for encoding "UTF8": 0xedbebf

BRUSSER Michael wrote:
> Is there a way to find the records with the text field containing Unicode bytes "0xedbebf"?
> Unfortunately this is a very old version 7.3.10

This should work on 7.3 (according to the documentation):
SELECT id FROM nlsdata WHERE position('\360\235\204\236'::bytea IN val::bytea) = 1;

--------------------------------------------------------------------------------

Albe, thanks for pointing this out!

I made a minor change, added decode since text cannot be cast to bytea and tried something like this:
SELECT id FROM myTable WHERE position('\360\235\204\236'::bytea IN decode(myTextField, 'escape')) != 0
ERROR: decode: Bad input string for type bytea

If I limit query to some healthy records - " AND id between 100 and 110" it works and returns empty result.
So the problem now is that without decode myTextField cannot be converted to bytea, with decode it breaks on the first 'bad' value.
Maybe this explains why?
testdb=# select decode('\360\235\204\236'::text, 'escape');
ERROR: Unicode >= 0x10000 is not supported

Another thought is that if I get this to work I may need to search for anything outside of the standard utf range,
rather than any specific sequence. I am beginning to understand why many people dealt with this in the dump file,
but I'm not ready to give up yet...

As usual, any ideas are appreciated!
Thanks.

This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.

If you are not one of the named recipients or have received this email in error,

(i) you should not read, disclose, or copy it,

(ii) please notify sender of your receipt by reply email and delete this email and all attachments,

(iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.

For other languages, go to http://www.3ds.com/terms/email-disclaimer