Re: [HACKERS] Thread-safe queueing?

Tim Holloway <mtsinc(at)southeast(dot)net> writes:
> Tom Lane wrote:
>> More context, please.

> This is for the logging subsystem I'm developing. The backends call
> pg_log(), which is like elog(), except that the message is a resource
> ID + any parameters in order to support locales and custom message
> formatting. These ID+parameter packets are then pipelined down to the
> logging channels via the log engine to be formatted and output
> according to rules in the configuration file.

OK. You probably want something roughly comparable to the shared-inval
message queue --- see include/storage/sinvaladt.h and
backend/storage/ipc/sinvaladt.c. That's more complex than your problem
in one way (the sinval queue must be read by all N backends, not just
one process) but simpler in another (we can shoehorn all SI messages
into a fixed-size structure; is that practical for log data?). Anyway,
a queue structure in shared memory protected by spinlocks is what you
want, and sinval is about the closest thing we have to that at the
moment.

> I *think* that the log engine should be a distinct process.

Probably so, if you use a shared-memory queue. Shared memory is a
finite resource; AFAIK it's not practical to enlarge it on-the-fly.
So you will have to set a maximum queue size --- either a compile-time
constant, or at best a value chosen at postmaster start time. This
implies that there will be scenarios where backends are waiting for room
to be made in the log queue. If the queue emptier is a separate process
then those waits can't turn into deadlocks. (sinval works around the
memory-overflow problem with a special "reset" mechanism, but that
doesn't seem appropriate for logging.)

Alternatively, you could forget about a queue per se, and just allow
each backend to execute the sending of its own log messages, using
a spinlock in shared memory to prevent concurrent issuance of log
messages on channels where that's a problem. That might be the
simplest and most robust approach.

regards, tom lane