Re: parallelizing the archiver

Greetings,

* Julien Rouhaud (rjuju123(at)gmail(dot)com) wrote:
> On Fri, Sep 10, 2021 at 2:03 PM Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
> > > 10 сент. 2021 г., в 10:52, Julien Rouhaud <rjuju123(at)gmail(dot)com> написал(а):
> > > Yes, but it also means that it's up to every single archiving tool to
> > > implement a somewhat hackish parallel version of an archive_command,
> > > hoping that core won't break it.

We've got too many archiving tools as it is, if you want my 2c on that.

> > I'm not proposing to remove existing archive_command. Just deprecate it one-WAL-per-call form.
>
> Which is a big API beak.

We definitely need to stop being afraid of this. We completely changed
around how restores work and made pretty much all of the backup/restore
tools have to make serious changes when we released v12.

I definitely don't think that we should be making assumptions that
changing archive command to start running things in parallel isn't
*also* an API break too, in any case. It is also a change and there's
definitely a good chance that it'd break some of the archivers out
there. If we're going to make a change here, let's make a sensible one.

> > It's a very simplistic approach. If some GUC is set - archiver will just feed ready files to stdin of archive command. What fundamental design changes we need?

Haven't really thought about this proposal but it does sound
interesting.

Thanks,

Stephen