Re: \COPY to accept non UTF-8 chars in CHAR columns

From: Andrew Gierth <andrew(at)tao11(dot)riddles(dot)org(dot)uk>
To: Matthias Apitz <guru(at)unixarea(dot)de>
Cc: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, "pgsql-generallists(dot)postgresql(dot)org" <pgsql-general(at)lists(dot)postgresql(dot)org>
Subject: Re: \COPY to accept non UTF-8 chars in CHAR columns
Date: 2020-03-28 14:08:47
Message-ID: 87bloga8ce.fsf@news-spur.riddles.org.uk
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

>>>>> "Matthias" == Matthias Apitz <guru(at)unixarea(dot)de> writes:

Matthias> i.e. 0xc3 is translated to 0xc383 and the 2nd half, the
Matthias> 0xbc to 0xc2bc, both translations have nothing to do with
Matthias> the original split 0xc3bc, and perhaps in this case it
Matthias> would be better to spill out a blank 0x40 for each of the
Matthias> bytes which formed the 0xc3bc.

If the only malformed sequences are there as a result of splitting up
valid sequences, then you could do something like convert all invalid
sequences to (sequences of) noncharacters, then once the data is
imported, fix it up by adjusting how the data is split and regenerating
the correct sequence (assuming your application allows this).

For example you could encode an arbitrary byte xy as a sequence of two
codepoints U+FDDx U+FDEy (the range FDD0-FDEF are all defined as
noncharacters).

--
Andrew (irc:RhodiumToad)

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Andrus 2020-03-28 15:18:50 Postgres 12 backup in 32 bit windows client
Previous Message Lucas Possamai 2020-03-28 12:28:52 PostegreSQL 9.2 to 9.6