Re: ArrayIndexOutOfBoundsException in Encoding.decodeUTF8()

Well this data was inserted into postgres through the jdbc driver in the first place.

So how come postgres itself didn't complain about non-ascii data? How do I change the
encoding? And what will the side effects be?

Barry Lind wrote:
> Joseph,
>
> The problem is that your database claims to be ASCII, but you are
> storing non-ascii data in it.
>
> As of 7.3 the jdbc driver has the server convert from the database
> character set to UTF8. Then send the data to the driver in UTF8 and the
> driver then decodes the UTF8 to java unicode.
>
> The conversion from ASCII to UTF8 is a noop since the 127 characters of
> ascii map directly to the same values in UTF8. However since you are
> storing not ASCII data the values that have the values from 128 - 255
> just get passed from the server to the client without any additional
> processing (since there aren't supposed to be any values in this range),
> but then when the driver tries to convert to java unicode, it can't
> because it has received an invalid UTF8 character.
>
> It seems that you are actually storing Latin1 data in this database and
> thus the database character set should probably be Latin1.
>
> In 7.2 is was possible to override the character set used by the driver,
> however I don't think this works anymore when connecting to a 7.3
> server. .... looks at code .... Yes the override is ignored if the
> server is a 7.3 server. You could hack at AbstractJdbc1Connection to
> work around the issue or just correctly set the database character set
> to match the data that the database contains.
>
> thanks,
> --Barry
>
>
> Joseph Shraibman wrote:
>
>> BTW the string that caused this is 'Oné'
>>
>> Joseph Shraibman wrote:
>>
>>> java.lang.ArrayIndexOutOfBoundsException: 3
>>> at org.postgresql.core.Encoding.decodeUTF8(Encoding.java:253)
>>> at org.postgresql.core.Encoding.decode(Encoding.java:165)
>>> at org.postgresql.core.Encoding.decode(Encoding.java:181)
>>> at
>>> org.postgresql.jdbc1.AbstractJdbc1ResultSet.getString(AbstractJdbc1ResultSet.java:97)
>>>
>>>
>>> The relavent code is:
>>>
>>> while (i < k) {
>>> z = data[i] & 0xFF;
>>> if (z < 0x80) {
>>> l_cdata[j++] = (char)data[i];
>>> i++;
>>> } else if (z >= 0xE0) { // length == 3
>>> y = data[i+1] & 0xFF; //<<== THIS IS LINE 253
>>> x = data[i+2] & 0xFF;
>>> val = (z-0xE0)*pow2_12 + (y-0x80)*pow2_6 + (x-0x80);
>>> l_cdata[j++] = (char) val;
>>> i+= 3;
>>> } else { // length == 2 (maybe add checking for
>>> length > 3, throw exception if it is
>>>
>>>
>>> And in the method that calls that:
>>>
>>> if (encoding.equals("UTF-8")) {
>>> return decodeUTF8(encodedString, offset, length);
>>> }
>>>
>>> The thing is my database encoding is SQL_ASCII
>>>
>>> => SELECT version(), getdatabaseencoding() ;
>>>
>>> version | getdatabaseencoding
>>> ---------------------------------------------------------------------------------------------------------+---------------------
>>>
>>> PostgreSQL 7.3.1 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.2
>>> 20020903 (Red Hat Linux 8.0 3.2-7) | SQL_ASCII
>>> (1 row)
>>>
>>> ... so why is it trying to decode the string as UTF-8? I just
>>> upgraded this database from 7.2.3 yesterday.
>>>
>>