Re: bg worker: overview

Hi,

On 07/15/2010 03:45 PM, Dimitri Fontaine wrote:
> We've been talking about this topic on -performance:

Thank for pointing out this discussion, I'm not following -performance
too closely.

> So, do you think we could use your work as a base for allowing custom
> daemon code?

Daemon code? That sounds like it could be an addition to the
coordinator, which I'm somewhat hesitant to extend, as it's a pretty
critical process (especially for Postgres-R).

With the step3, which adds support for sockets, you can use the
coordinator to listen on pretty much any kind of socket you want. That
might be helpful in some cases (just as it is required for connecting to
the GCS).

However, note that the coordinator is designed to be just a message
passing or routing process, which should not do any kind of time
consuming processing. It must *coordinate* things (well, jobs) and react
promptly. Nothing else.

On the other side, the background workers have a connection to exactly
one database. They are supposed to do work on that database.

> I guess we need to think about how to separate external
> code and internal code, so a second layer could be necessary here.

The background workers can easily load external libraries - just as a
normal backend can with LOAD. That would also provide better
encapsulation (i.e. an error would only tear down that backend, not the
coordinator). You'd certainly have to communicate between the
coordinator and the background worker. I'm not sure how match that fits
your use case.

The thread on -performance is talking quite a bit about connection
pooling. The only way I can imagine some sort of connection pooling to
be implemented on top of bgworkers would be to let the coordinator
listen on an additional port and pass on all requests to the bgworkers
as jobs (using imessages). And of course send back the responses to the
client. I'm not sure how that overhead compares to using pgpool or
pgbouncer. Those are also separate processes through which all of your
data must flow. They use plain system sockets, imessages use signals and
shared memory.

I don't know enough about the pgagent or PgQ use cases to comment,
sorry. Hope that's helpful, anyway.

Regards

Markus