Re: Send duration output to separate log files

On Thu, Jul 11, 2024 at 6:47 AM Alastair Turner <minion(at)decodable(dot)me> wrote:

> The other category of logging which would benefit from a separate file is
> audit. It also can create massive volumes of log content. Splitting audit
> information off into a separate file for use by a separate team or function
> is also a request I have heard from some financial institutions adopting
> Postgres. With audit being provided by an extension, this would become
> quite an intrusive change.
>

Thanks for the feedback. I agree pgaudit is another thing that can create
massive log files, and should be solved at some point. However, I wanted
to keep this patch self-contained to in-core stuff. And pgaudit is already
an odd duck, in that it puts CSV into your stderr stream (and into your
json!). Ideally it would put a single CSV stream into a separate csv file.
Perhaps even something that did not necessarily live in log_directory.

Would an extension be able to safely modify the message_type field you have
>> added using emit_log_hook? If so, the field becomes more of a log
>> destination label than a type marker. If an extension could hook into the
>> log file creation/rotation process, that would be a nice basis for enabling
>> extensions (I'm particularly thinking of pgAudit) to manage separate
>> logging destinations.
>>
>
Yes, I had more than duration in mind when I created errmessagetype. A hook
to set it would be the obvious next step, and then some sort of way of
mapping that to arbitrary log files. But I see that as mostly orthagonal to
this patch (and certainly a much larger endeavor).

(wades in anyways). I'm not sure about hooking into the log rotation
process so much as registering something on startup, then letting Postgres
handle all the log files in the queue. Although as I alluded to above,
sometimes having large log files NOT live in the data directory (or more
specifically, not hang out with the log_directory crowd), could be a plus
for space, efficiency, and security reasons. That makes log rotation
harder, however. And do we / should we put extension-driven logfiles into
current_logfiles? Do we still fall back to stderr even for extension logs?
Lots to ponder. :)

Cheers,
Greg