Re: logging in high performance systems.

On Wed, Nov 23, 2011 at 9:28 PM, Theo Schlossnagle <jesus(at)omniti(dot)com> wrote:
> We have a need for logging in systems where it isn't feasible to log
> to disk as it negatively impacts performance.
>
> I'd like to be able to creatively solve this problem without modifying
> the core, but today I cannot.
>
> So... here's my first whack at solving this with some flexibility.
>
> The first thing I did was add hook points where immediate statement
> logging happens "pre_exec" and those that present duration
> "post_exec".  These should, with optimization turned on, have only a
> few instructions of impact when no hooks are registered (we could
> hoist the branch outside the function call if that were identified as
> an issue).
>
> https://github.com/postwait/postgres/commit/62bb9dfa2d373618f10e46678612720a3a01599a
>
> The second thing I did was write a sample use of those hooks to
> implement a completely non-blocking fifo logger. (if it would block,
> it drops the log line).  The concept is that we could run this without
> risk of negative performance impact due to slow log reading (choosing
> to drop logs in lieu of pausing).  And a simple process could be
> written to consume from the fifo.  We use this method in other systems
> to log many 10s of thousands of log lines per second with negligible
> impact on performance.
>
> https://github.com/postwait/postgres/commit/c8f5a346c7b2c3eba9f72ea49077dc72f03a2679
>
> Thoughts? Feedback?
>
> As can be seen, the patch is pretty tiny.

Theo,

Tom recently committed something by another author that is along
similar lines to what you have here (I think). Can you comment on
whether you think more is still needed and what the differences are
between that approach and yours?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company