Re: On Logging

On 9/26/05, David Fetter <david(at)fetter(dot)org> wrote:
> I've run into something that concerns me. It's pretty much an 8.2
> issue, but I'm hoping to stimulate some discussion on it. It's
> PostgreSQL's log files. Right now, they're (sometimes just barely ;)
> human-readable, but they take significant effort to parse. For
> example, pqa, a very clever piece of code, is mostly devoted to
> parsing said files and works only with significant tweaking and
> restrictions on log file formats in 8.0.

In a previous life (oh, like 6 months ago), I spent all my time
working on parsing log files from dozens of different software
products, and I learned something that made parsing some files orders
of magnitude easier than others:

Always use message codes.

Cisco does this, and it helps a lot. A few other vendors do this, and
it helps a lot. While this might seem an old mainframeism, it's
terribly useful to have something at the beginning that tells you what
the message is, what it means, and most importantly, how to parse the
rest.

I would be happy to help create this catalog, though it's definately a
big step to implement. It would also require identifying every message
that could be generated -- something few open source projects do, but
it is critical to those of us who have to process the output!

> Simple logging is a default that should probably not change, but I'm
> thinking that for people who want to find something out from the logs,
> we could see about a kind of plugin architecture which would enable
> things like:
>
> * CSV

CSV is the best format, ever. Trivially simple to parse, it requires
no extra processing so long as you abide by a few extra rules, such as
escaping.

> * YAML

Nice, but I think perhaps not the best format for logging. It's more
of a configuration file format in my mind, and it requires a bit more
oompf to parse. Not going to happen in AWK. :-)

> * Piped logs, as Apache can do

Useful, but doesn't create any new capabilities, just simplifies some
of them. Focus on "new capabilities" first, then added functionality
if required.

> * DB handle. I know this one will be controversial.

I can't imagine why. :-)

> 1. Am I the only one who would wants an option for machine-readable logs?

Not likely. I'd love it. It makes monitoring and reporting easier.

Chris
--
| Christopher Petrilli
| petrilli(at)gmail(dot)com