[v9.3] Extra Daemons (Re: elegant and effective way for running jobs inside a database)

2012/3/10 Simon Riggs <simon(at)2ndquadrant(dot)com>:
> On Fri, Mar 9, 2012 at 6:51 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>>
>>
>> On 03/09/2012 01:40 PM, Robert Haas wrote:
>>>
>>> On Fri, Mar 9, 2012 at 12:02 PM, David E. Wheeler<david(at)justatheory(dot)com>
>>>  wrote:
>>>>
>>>> On Mar 9, 2012, at 7:55 AM, Merlin Moncure wrote:
>>>>>
>>>>> 100% agree  (having re-read the thread and Alvaro's idea having sunk
>>>>> in).  Being able to set up daemon processes side by side with the
>>>>> postmaster would fit the bill nicely.  It's pretty interesting to
>>>>> think of all the places you could go with it.
>>>>
>>>> pgAgent could use it *right now*. I keep forgetting to restart it after
>>>> restarting PostgreSQL and finding after a day or so that no jobs have run.
>>>
>>> That can and should be fixed by teaching pgAgent that failing to
>>> connect to the server, or getting disconnected, is not a fatal error,
>>> but a reason to sleep and retry.
>>
>>
>> Yeah. It's still not entirely clear to me what a postmaster-controlled
>> daemon is going to be able to do that an external daemon can't.
>
> Start and stop at the same time as postmaster, without any pain.
>
> It's a considerable convenience to be able to design this aspect once
> and then have all things linked to the postmaster follow that. It
> means people will be able to write code that runs on all OS easily,
> without everybody having similar but slightly different code about
> starting up, reading parameters, following security rules etc.. Tight
> integration, with good usability.
>
I tried to implement a patch according to the idea. It allows extensions
to register an entry point of the self-managed daemon processes,
then postmaster start and stop them according to the normal manner.

[kaigai(at)iwashi patch]$ ps ax | grep postgres
27784 pts/0 S 0:00 /usr/local/pgsql/bin/postgres
27786 ? Ss 0:00 postgres: writer process
27787 ? Ss 0:00 postgres: checkpointer process
27788 ? Ss 0:00 postgres: wal writer process
27789 ? Ss 0:00 postgres: autovacuum launcher process
27790 ? Ss 0:00 postgres: stats collector process
27791 ? Ss 0:00 postgres: auth_counter <== (*)

The auth_counter being included in this patch is just an example of
this functionality. It does not have significant meanings. It just logs
number of authentication success and fails every intervals.

I'm motivated to define an extra daemon that attach shared memory
segment of PostgreSQL as a computing server to avoid limitation of
number of GPU code that we can load concurrently.

Thanks,
--
KaiGai Kohei <kaigai(at)kaigai(dot)gr(dot)jp>