Re: .ready and .done files considered harmful

On 9/24/21 12:28 PM, Robert Haas wrote:
> On Thu, Sep 16, 2021 at 7:26 PM Bossart, Nathan <bossartn(at)amazon(dot)com> wrote:
>> What do you think?
>
> I think this is committable. I also went back and looked at your
> previous proposal to do files in batches, and I think that's also
> committable. After some reflection, I think I have a slight preference
> for the batching approach.
> It seems like it might lend itself to archiving multiple files in a
> single invocation of the archive_command, and Alvaro just suggested it
> again apparently not having realized that it had been previously
> proposed by Andres, so I guess it has the further advantage of being
> the thing that several committers intuitively feel like we ought to be
> doing to solve this problem.

I also prefer this approach. Reducing directory scans is an excellent
optimization, but from experience I know that execution time for the
archive_command can also be a significant bottleneck. Begin able to
archive multiple segments per execution would be a big win in certain
scenarios.

> So what I am inclined to do is commit
> v1-0001-Improve-performance-of-pgarch_readyXlog-with-many.patch.

I read the patch and it looks good to me.

I do wish we had a way to test that history files get archived first,
but as I recall I was not able to figure out how to do reliably for [1]
without writing a custom archive_command just for testing. That is
something we might want to consider as we make this logic more complex.

Regards,
--
-David
david(at)pgmasters(dot)net

[1]
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=b981df4cc09aca978c5ce55e437a74913d09cccc