Re: On Logging

On Mon, Sep 26, 2005 at 01:13:08PM -0400, Christopher Petrilli wrote:
> On 9/26/05, David Fetter <david(at)fetter(dot)org> wrote:
> > I've run into something that concerns me. It's pretty much an 8.2
> > issue, but I'm hoping to stimulate some discussion on it. It's
> > PostgreSQL's log files. Right now, they're (sometimes just barely
> > ;) human-readable, but they take significant effort to parse. For
> > example, pqa, a very clever piece of code, is mostly devoted to
> > parsing said files and works only with significant tweaking and
> > restrictions on log file formats in 8.0.
>
> In a previous life (oh, like 6 months ago), I spent all my time
> working on parsing log files from dozens of different software
> products, and I learned something that made parsing some files
> orders of magnitude easier than others:
>
> Always use message codes.

Could you elucidate a bit on this as to how this might affect
PostgreSQL logging?

> Cisco does this, and it helps a lot. A few other vendors do this,
> and it helps a lot. While this might seem an old mainframeism,
^^^^^^^^^^^^^^^^^
You say that like it's a *bad* thing. I think some fruitful
communication is possible and has been missed over the decades between
mainframe people and *n*x people. The same applies to supercomputing
people and *n*x people, but that's a story for another day.

> it's terribly useful to have something at the beginning that tells
> you what the message is, what it means, and most importantly, how to
> parse the rest.

OK

> I would be happy to help create this catalog, though it's definately
> a big step to implement. It would also require identifying every
> message that could be generated -- something few open source
> projects do, but it is critical to those of us who have to process
> the output!

Right. How big a project is this, and what kind of framework would we
need in order assure that new messages come with new message codes?

> > Simple logging is a default that should probably not change, but
> > I'm thinking that for people who want to find something out from
> > the logs, we could see about a kind of plugin architecture which
> > would enable things like:
> >
> > * CSV
>
> CSV is the best format, ever. Trivially simple to parse, it
> requires no extra processing so long as you abide by a few extra
> rules, such as escaping.

I agree that it's nice, but seeing as how many smart people have
stubbed their toes on the various incarnations of "CSV," I must
disagree as to its simplicity.

> > * YAML
>
> Nice, but I think perhaps not the best format for logging. It's
> more of a configuration file format in my mind, and it requires a
> bit more oompf to parse. Not going to happen in AWK. :-)

It's not bad for logging, partly because it's a lot fewer bytes than
XML or SGML, but it maintains a structure. Of course, it's not as
"simple" in some sense as CSV.

> > * Piped logs, as Apache can do
>
> Useful, but doesn't create any new capabilities, just simplifies
> some of them. Focus on "new capabilities" first, then added
> functionality if required.

Fair enough :)

> > * DB handle. I know this one will be controversial.
>
> I can't imagine why. :-)

Heh

> > 1. Am I the only one who would wants an option for machine-readable
> > logs?
>
> Not likely. I'd love it. It makes monitoring and reporting easier.

That's where I've run across this :)

Cheers,
D
--
David Fetter david(at)fetter(dot)org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!