Re: Using read stream in autoprewarm

Hi,

Thank you for looking into this!

On Wed, 27 Nov 2024 at 16:50, Matheus Alcantara <mths(dot)dev(at)pm(dot)me> wrote:
> I've executed the same test of 5 databases with each of them having 1 table of
> 3GB of size and I've got very similar results.
>
> I've also tested using a single database with 4 tables with ~60GB of size and
> the results compared with master was more closer but still an improvement. Note
> that I've also increased the default shared_buffers to 7GB to see how it works
> with large buffer pools.
> - patched: 5.4259 s
> - master: 5.53186 s

Thanks for the testing.

> Not to much to say about the code, I'm currently learning more about the read
> stream API and Postgresql hacking itself. Just some minor points and questions
> about the patches.
>
>
> v2-0002-Count-free-buffers-at-the-start-of-the-autoprewar.patch
> --- a/src/backend/storage/buffer/freelist.c
> +/*
> + * get_number_of_free_buffers -- a lockless way to get the number of free
> + * buffers in buffer pool.
> + *
> + * Note that result continuosly changes as free buffers are moved out by other
> + * operations.
> + */
> +int
> +get_number_of_free_buffers(void)
>
> typo on continuosly -> continuously

Done.

> v2-0001-Use-read-stream-in-autoprewarm.patch
> + bool *rs_have_free_buffer = per_buffer_data;
> +
> +
> + *rs_have_free_buffer = true;
> +
>
> Not sure if I understand why this variable is needed, it seems that it is only
> written and never read? Just as comparison, the block_range_read_stream_cb
> callback used on pg_prewarm seems to not use the per_buffer_data parameter.

Actually, it is read in the main loop of the
autoprewarm_database_main() function:

/* There are no free buffers left in shared buffers, break the loop. */
else if (!(*rs_have_free_buffer))
break;

apw_read_stream_next_block() callback function sets
rs_have_free_buffer's value to false when there is no free buffer left
in the shared buffers. And the code above terminates the main loop in
the autoprewarm_database_main() function when it is set to false.

block_range_read_stream_cb() callback is used when the callback only
needs to loop over the block numbers. However, for the autoprewarm
case; the callback function needs to do additional checks so another
callback and the use of this variable are required.

v3 is attached.

--
Regards,
Nazir Bilal Yavuz
Microsoft