Re: Parallel heap vacuum

On Sun, Mar 23, 2025 at 10:13 AM Andres Freund <andres(at)anarazel(dot)de> wrote:
>
> Hi,
>
> On 2025-03-23 01:45:35 -0700, Masahiko Sawada wrote:
> > Another idea is that parallel workers don't exit phase 1 until it
> > consumes all pinned buffers in the queue, even if the memory usage of
> > TidStore exceeds the limit.
>
> Yes, that seems a quite reasonable approach to me.
>
>
> > It would need to add new functionality to the read stream to disable the
> > look-ahead reading.
>
> Couldn't your next block callback simply return InvalidBlockNumber once close
> to the memory limit?

Good idea.

>
>
> > Since we could use much memory while processing these buffers, exceeding the
> > memory limit, we can trigger this mode when the memory usage of TidStore
> > reaches 70% of the limit or so.
>
> It wouldn't be that much memory, would it? A few 10s-100s of buffers don't
> increase the size of a TidStore that much? Using 10 parallel vacuum with a
> m_w_m of 1MB doesn't make sense, we'd constantly start/stop workers.
>

It ultimately depends on how big the next size class of DSA segment
is, but typically yes. We configure the max DSA segment size based on
the maintenance_work_mem.

>
> > On the other hand, it means that we would not use the streaming read for the
> > blocks in this mode, which is not efficient.
>
> I don't follow - why wouldn't you be using streaming read?

I was thinking of idea like stopping look-ahead reading at some point
and processing pages without the read stream until the TidStore
reaches the limit. But it seems like simpler if we could stop
retrieving next block for the read stream once the TidStore reached
the limit and then continue phase 1 until the read stream is
exhausted.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com