Re: Postgres-R: internal messaging

Markus Wanner wrote:
> Besides the communication between the replication manager and the
> backends, which is currently done by using these imessages, the
> replication manager also needs to communicate with the postmaster: it
> needs to be able to request new helper backends and it wants to be
> notified upon termination (or crash) of such a helper backend (and other
> backends as well...). I'm currently doing this with imessages as well,
> which violates the rule that the postmaster may not to touch shared
> memory. I didn't look into ripping that out, yet. I'm not sure it can be
> done with the existing signaling of the postmaster.

In Replicator we avoided the need for postmaster to read/write backend's
shmem data by using it as a signal forwarder. When a backend wants to
inform a special process (i.e. queue monitor) about replication-related
event (such as commit) it sends SIGUSR1 to Postmaster with a related
"reason" flag and the postmaster upon receiving this signal forwards
it to the destination process. Termination of backends and special
processes are handled by the postmaster itself.

>
> Let's have a simple example: consider a local transaction which changes
> some tuples. Those are being collected into a change set, which gets
> written to the shared memory area as an imessage for the replication
> manager. The backend then also signals the manager, which then awakes
> from its select(), checks its imessages queue and processes the message,
> delivering it to the GCS. It then removes the imessage from the shared
> memory area again.

Hm...what would happen with the new data under heavy load when the queue
would eventually be filled with messages, the relevant transactions
would be aborted or they would wait for the manager to release the queue
space occupied by already processed messages? ISTM that having a fixed
size buffer limits the maximum transaction rate.

>
> My initial design features only a single doubly linked list as the
> message queue, holding all messages for all processes. An imessages lock
> blocks concurrent writing acces. That's still what's in there, but I
> realize that's not enough. Each process should better have its own
> queue, and the single lock needs to vanish to avoid contention on that
> lock. However, that would require dynamically allocatable shared
> memory...

What about keeping the per-process message queue in the local memory of
the process, and exporting only the queue head to the shmem, thus having
only one message per-process there. When the queue manager gets a
message from the process it may signal that process to copy the next
message from the process local memory into the shmem. To keep a
correct ordering of queue messages an additional shared memory queue of
pid_t can be maintained, containing one pid per each message.

--
Alexey Klyukin http://www.commandprompt.com/
The PostgreSQL Company - Command Prompt, Inc.