Re: parallelizing the archiver

On Tue, Oct 19, 2021 at 2:50 PM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Mon, Oct 18, 2021 at 7:25 PM Bossart, Nathan <bossartn(at)amazon(dot)com>
> wrote:
> > I think the biggest question is where to put the archive_command
> > module, which I've called shell_archive. The only existing directory
> > that looked to me like it might work is src/test/modules. It might be
> > rather bold to relegate this functionality to a test module so
> > quickly, but on the other hand, perhaps it's the right thing to do
> > given we intend to deprecate it in the future. I'm curious what
> > others think about this.
>
> I don't see that as being a viable path forward based on my customer
> interactions working here at EDB.
>
> I am not quite sure why we wouldn't just compile the functions into
> the server. Functions pointers can point to core functions as surely
> as loadable modules. The present design isn't too congenial to that
> because it's relying on the shared library loading mechanism to wire
> the thing in place - but there's no reason it has to be that way.
> Logical decoding plugins don't work that way, for example. We could
> still have a GUC, say call it archive_method, that selects the module
> -- with 'shell' being a builtin method, and others being loadable as
> modules. If you set archive_method='shell' then you enable this
> module, and it has its own GUC, say call it archive_command, to
> configure the behavior.
>

Yeah, seems reasonable. It wouldn't serve as well as an example to
developers, but then it's probably not the "loadable module" part of
building it that people need examples of. So as long as it's using the same
internal APIs and just happens to be compiled in by default, I see no
problem with that.

But, is logical decoding really that great an example? I mean, we build
pgoutput.so as a library, we don't provide it compiled-in. So we could
build the "shell archiver" based on that pattern, in which case we should
create a postmaster/shell_archiver directory or something like that?

It should definitely not go under "test".

An advantage of this approach is that it's perfectly
> backward-compatible. I understand that archive_command is a hateful
> thing to many people here, but software has to serve the user base,
> not just the developers. Lots of people use archive_command and rely
> on it -- and are not interested in installing yet another piece of
> out-of-core software to do what $OTHERDB has built in.
>

Backwards compatibility is definitely a must, I'd say. Regardless of
exactly how the backwards-compatible pugin is shipped, it should be what's
turned on by default.

--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/>
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>