Re: BUG #5661: The character encoding in logfile is confusing.

On 22/09/2010 5:45 PM, Peter Eisentraut wrote:
> On ons, 2010-09-22 at 16:25 +0800, Craig Ringer wrote:
>> A single log file should obviously be in a single encoding, it's the
>> only sane way to do things. But which encoding is it in? And which
>> *should* it be in?
>
> We need to produce the log output in the server encoding, because that's
> how we need to send it to the client.

That doesn't mean it can't be recoded for writing to the log file,
though. Perhaps it needs to be. It should be reasonably practical to
detect when the database and log encoding are the same and avoid the
transcoding performance penalty, not that it's big anyway.

> If you have different databases
> with different server encodings, you will get inconsistently encoded
> output in the log file.

I don't think that's an OK answer, myself. Mixed encodings with no
delineation in one file = bug as far as I'm concerned. You can't even
rely on being able to search the log anymore. You'll only get away with
it when using languages that mostly stick to the 7-bit ASCII subset, so
most text is still readable; with most other languages you'll get logs
full of what looks to the user like garbage.

> Conceivably, we could create a configuration option that specifies the
> encoding for the log file, and strings a recoded from whatever gettext()
> produces to the specified encoding. initdb could initialize that option
> suitably, so in most cases users won't have to do anything.

Yep, I tend to think that'd be the right way to go. It'd still be a bit
of a pain, though, as messages written to stdout/stderr by the
postmaster should be in the system encoding, but messages written to the
log files should be in the encoding specified for logs, unless logging
is being done to syslog, in which case it has to be in the system
encoding after all...

And, of course, the postmaster still doesn't know how to log anything it
might emit before reading postgresql.conf, because it doesn't know what
encoding to use.

I still wonder if, rather than making this configurable, the right
choice is to force logging to UTF-8 (with BOM) across the board, right
from postmaster startup. It's consistent, all messages in all other
encodings can be converted to UTF-8 for logging, it's platform
independent, and text editors etc tend to understand and recognise UTF-8
especially with the BOM.

Unfortunately, because many unix utilities (grep etc) aren't encoding
aware, that'll cause problems when people go to search log files. For
(eg) "広告掲載" The log files will contain the utf-8 bytes:

\xe5\xba\x83\xe5\x91\x8a\xe6\x8e\xb2\xe8\xbc\x89

but grep on a shift-jis system will be looking for:

\x8d\x4c\x8d\x90\x8cf\x8d\xda

so it won't match.

Ugh. If only we could say "PostgreSQL requires a system locale with a
UTF-8 encoding". Alas, I don't think that'd go down very well with
packagers or installers. [Insert rant about how stupid it is that *nix
systems still aren't all UTF-8 here].

--
Craig Ringer

Tech-related writing at http://soapyfrogs.blogspot.com/