Re: parallelizing the archiver

On 10/19/21 8:50 AM, Robert Haas wrote:
> On Mon, Oct 18, 2021 at 7:25 PM Bossart, Nathan <bossartn(at)amazon(dot)com> wrote:
>> I think the biggest question is where to put the archive_command
>> module, which I've called shell_archive. The only existing directory
>> that looked to me like it might work is src/test/modules. It might be
>> rather bold to relegate this functionality to a test module so
>> quickly, but on the other hand, perhaps it's the right thing to do
>> given we intend to deprecate it in the future. I'm curious what
>> others think about this.
>
> I don't see that as being a viable path forward based on my customer
> interactions working here at EDB.
>
> I am not quite sure why we wouldn't just compile the functions into
> the server. Functions pointers can point to core functions as surely
> as loadable modules. The present design isn't too congenial to that
> because it's relying on the shared library loading mechanism to wire
> the thing in place - but there's no reason it has to be that way.
> Logical decoding plugins don't work that way, for example. We could
> still have a GUC, say call it archive_method, that selects the module
> -- with 'shell' being a builtin method, and others being loadable as
> modules. If you set archive_method='shell' then you enable this
> module, and it has its own GUC, say call it archive_command, to
> configure the behavior.
>
> An advantage of this approach is that it's perfectly
> backward-compatible. I understand that archive_command is a hateful
> thing to many people here, but software has to serve the user base,
> not just the developers. Lots of people use archive_command and rely
> on it -- and are not interested in installing yet another piece of
> out-of-core software to do what $OTHERDB has built in.

+1 to all of this, certainly for the time being. The archive_command
mechanism is not great, but it is simple, and this part is not really
what makes writing a good archive command hard.

I had also originally envisioned this a default extension in core, but
having the default 'shell' method built-in is certainly simpler.

Regards,
--
-David
david(at)pgmasters(dot)net