7.2. Multibyte Support

7.2.1. Supported character set encodings

Following encoding can be used as database encoding.

7.2.2. Setting the Encoding

initdb defines the default encoding for a PostgreSQL installation. For example:

$ initdb -E EUC_JP

sets the default encoding to EUC_JP (Extended Unix Code for Japanese). Note that you can use --encoding instead of -E if you prefer to type longer option strings. If no -E or --encoding option is given, SQL_ASCII is used.

You can create a database with a different encoding:

$ createdb -E EUC_KR korean

will create a database named korean with EUC_KR encoding. Another way to accomplish this is to use a SQL command:

CREATE DATABASE korean WITH ENCODING = 'EUC_KR';

The encoding for a database is represented as an encoding column in the pg_database system catalog. You can see that by using the -l option or the \l command of psql.

$ psql -l
            List of databases
   Database    |  Owner  |   Encoding    
---------------+---------+---------------
 euc_cn        | t-ishii | EUC_CN
 euc_jp        | t-ishii | EUC_JP
 euc_kr        | t-ishii | EUC_KR
 euc_tw        | t-ishii | EUC_TW
 mule_internal | t-ishii | MULE_INTERNAL
 regression    | t-ishii | SQL_ASCII
 template1     | t-ishii | EUC_JP
 test          | t-ishii | EUC_JP
 unicode       | t-ishii | UNICODE
(9 rows)

7.2.3. Automatic encoding conversion between server and client

PostgreSQL supports an automatic encoding conversion between server and client for some encodings. The conversion info is stored in pg_conversion system catalog. You can create a new conversion by using CREATE CONVERSION. PostgreSQL comes with some predefined conversions. They are listed in Table 7-2.

To enable the automatic encoding translation, you have to tell PostgreSQL the encoding you would like to use in the client. There are several ways to accomplish this.

• Using the \encoding command in psql. \encoding allows you to change client encoding on the fly. For example, to change the encoding to SJIS, type:

  \encoding SJIS
  

• Using libpq functions. \encoding actually calls PQsetClientEncoding() for its purpose.

  int PQsetClientEncoding(PGconn *conn, const char *encoding)
  

  where conn is a connection to the server, and encoding is an encoding you want to use. If it successfully sets the encoding, it returns 0, otherwise -1. The current encoding for this connection can be shown by using:

  int PQclientEncoding(const PGconn *conn)
  

  Note that it returns the encoding ID, not a symbolic string such as EUC_JP. To convert an encoding ID to an encoding name, you can use:

  char *pg_encoding_to_char(int encoding_id)
  

• Using SET CLIENT_ENCODING TO. Setting the client encoding can be done with this SQL command:

  SET CLIENT_ENCODING TO 'encoding';
  

  Also you can use the SQL92 syntax SET NAMES for this purpose:

  SET NAMES 'encoding';
  

  To query the current client encoding:

  SHOW CLIENT_ENCODING;
  

  To return to the default encoding:

  RESET CLIENT_ENCODING;
  

• Using PGCLIENTENCODING. If environment variable PGCLIENTENCODING is defined in the client's environment, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.)

• Using client_encoding variable. If the client_encoding variable in postgresql.conf is set, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.)

7.2.4. What happens if the translation is not possible?

Suppose you choose EUC_JP for the server and LATIN1 for the client, then some Japanese characters cannot be translated into LATIN1. In this case, a letter that cannot be represented in the LATIN1 character set would be transformed as:

(HEXA DECIMAL)

7.2.5. References

These are good sources to start learning about various kinds of encoding systems.

7.2.6. History

Dec 7, 2000
        * An automatic encoding translation between Unicode and other
          encodings are implemented
        * Changes above will appear in 7.1

May 20, 2000
        * SJIS UDC (NEC selection IBM kanji) support contributed
          by Eiji Tokuya
        * Changes above will appear in 7.0.1

Mar 22, 2000
        * Add new libpq functions PQsetClientEncoding, PQclientEncoding
        * ./configure --with-mb=EUC_JP
          now deprecated. use 
          ./configure --enable-multibyte=EUC_JP
          instead
        * Add SQL_ASCII regression test case
        * Add SJIS User Defined Character (UDC) support
        * All of above will appear in 7.0

July 11, 1999
        * Add support for WIN1250 (Windows Czech) as a client encoding
          (contributed by Pavel Behal)
        * fix some compiler warnings (contributed by Tomoaki Nishiyama)

Mar 23, 1999
        * Add support for KOI8(KOI8-R), WIN(CP1251), ALT(CP866)
          (thanks Oleg Broytmann for testing)
        * Fix problem with MB and locale

Jan 26, 1999
        * Add support for Big5 for frontend encoding
          (you need to create a database with EUC_TW to use Big5)
        * Add regression test case for EUC_TW
          (contributed by Jonah Kuo <jonahkuo@mail.ttn.com.tw>)

Dec 15, 1998
        * Bugs related to SQL_ASCII support fixed

Nov 5, 1998
        * 6.4 release. In this version, pg_database has "encoding"
          column that represents the database encoding

Jul 22, 1998
        * determine encoding at initdb/createdb rather than compile time
        * support for PGCLIENTENCODING when issuing COPY command
        * support for SQL92 syntax "SET NAMES"
        * support for LATIN2-5
        * add UNICODE regression test case
        * new test suite for MB
        * clean up source files

Jun 5, 1998
        * add support for the encoding translation between the backend
          and the frontend
        * new command SET CLIENT_ENCODING etc. added
        * add support for LATIN1 character set
        * enhance 8-bit cleanliness

April 21, 1998 some enhancements/fixes
        * character_length(), position(), substring() are now aware of 
          multi-byte characters
        * add octet_length()
        * add --with-mb option to configure
        * new regression tests for EUC_KR
          (contributed by Soonmyung Hong)
        * add some test cases to the EUC_JP regression test
        * fix problem in regress/regress.sh in case of System V
        * fix toupper(), tolower() to handle 8bit chars

Mar 25, 1998 MB PL2 is incorporated into PostgreSQL 6.3.1

Mar 10, 1998 PL2 released
        * add regression test for EUC_JP, EUC_CN and MULE_INTERNAL
        * add an English document (this file)
        * fix problems concerning 8-bit single byte characters

Mar 1, 1998 PL1 released

7.2.7. WIN1250 on Windows/ODBC

The WIN1250 character set on Windows client platforms can be used with PostgreSQL with locale support enabled.

The following should be kept in mind:

• Success depends on proper system locales. This has been tested with Red Hat 6.0 and Slackware 3.6, with the cs_CZ.iso8859-2 locale.

• Never try to set the server's database encoding to WIN1250. Always use LATIN2 instead since there is no WIN1250 locale in Unix.

• The WIN1250 encoding is usable only for Windows ODBC clients. The characters are recoded on the fly, to be displayed and stored back properly.