Re: Suggestions on message transfer among backends

On Tue, Mar 12, 2019 at 4:34 AM Antonin Houska <ah(at)cybertec(dot)at> wrote:
> Andy Fan <zhihui(dot)fan1213(at)gmail(dot)com> wrote:
> > I just don't know why shm_mq is designed to single-reader & single-writer.
>
> shm_mq was implemented as a part of infrastructure for parallel query
> processing. The leader backend launches multiple parallel workers and sets up
> a few queues to communicate with each. One queue is used to send request
> (query plan) to the worker, one queue is there to receive data from it, and I
> think there's one more queue to receive error messages.

No, the queues aren't used to send anything to the worker. We know
the size of the query plan before we create the DSM, so we can just
allocate enough space to store the whole thing. We don't know the
size of the result set, though, so we use a queue to retrieve that
from the worker. And we also don't know the size of any warnings or
errors or other such things that the worker might generate, so we use
a queue to retrieve that stuff, too. It turned out to be better to
have a separate queue for each of those things rather than a single
queue for both.

I admit that I could have design a system that supported multiple
readers and writers and that it would have been useful, but it also
would have been more work, and there's something to be said for
finishing the feature before your boss fires you. Also, such a system
would probably have more overhead; shm_mq can do a lot of things
without locks that would need locks if you had multiple readers and
writers.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company