Re: Weird failure with latches in curculio on v15

On Wed, Feb 8, 2023 at 7:24 PM Nathan Bossart <nathandbossart(at)gmail(dot)com> wrote:
> On Thu, Feb 09, 2023 at 08:56:24AM +0900, Michael Paquier wrote:
> > On Wed, Feb 08, 2023 at 02:25:54PM -0800, Nathan Bossart wrote:
> >> These are all good points. Perhaps there could be a base archiver
> >> implementation that shell_archive uses (and that other modules could use if
> >> desired, which might be important for backward compatibility with the
> >> existing callbacks). But if you want to do something fancier than
> >> archiving sequentially, you could write your own.
> >
> > Which is basically the kind of things you can already achieve with a
> > background worker and a module of your own?
>
> IMO one of the big pieces that's missing is a way to get the next N files
> to archive. Right now, you'd have to trawl through archive_status on your
> own if you wanted to batch/parallelize. I think one advantage of what
> Robert is suggesting is that we could easily provide a supported way to get
> the next set of files to archive, and we can asynchronously mark them
> "done". Otherwise, each module has to implement this.

Right.

I think that we could certainly, as Michael suggests, have people
provide their own background worker rather than having the archiver
invoke the user-supplied code directly. As long as the functions that
you need in order to get the necessary information can be called from
some other process, that's fine. The only difficulty I see is that if
the archiving is happening from a separate background worker rather
than from the archiver, then what is the archiver doing? We could
somehow arrange to not run the archiver process at all, or I guess to
just sit there and have it do nothing. Or, we can decide not to have a
separate background worker and just have the archiver call the
user-supplied core directly. I kind of like that approach at the
moment; it seems more elegant to me.

--
Robert Haas
EDB: http://www.enterprisedb.com