Re: refactoring basebackup.c

On Fri, Oct 15, 2021 at 7:54 AM Jeevan Ladhe
<jeevan(dot)ladhe(at)enterprisedb(dot)com> wrote:
> > The loop is gone, but CHUNK_SIZE itself seems to have evaded the executioner.
>
> I am sorry, but I did not really get it. Or it is what you have pointed
> in the following paragraphs?

I mean #define CHUNK_SIZE is still in the patch.

> > With just that change, you can set
> > next_buf_len = LZ4F_HEADER_SIZE_MAX + mysink->output_buffer_bound --
> > but that's also more than you need. You can instead do next_buf_len =
> > Min(LZ4F_HEADER_SIZE_MAX, mysink->output_buffer_bound). Now, you're
> > probably thinking that won't work, because bbsink_lz4_begin_archive()
> > could fill up the buffer partway, and then the first call to
> > bbsink_lz4_archive_contents() could overrun it. But that problem can
> > be solved by reversing the order of operations in
> > bbsink_lz4_archive_contents(): before you call LZ4F_compressUpdate(),
> > test whether you need to empty the buffer first, and if so, do it.
>
> I am still not able to get - how can we survive with a mere
> size of Min(LZ4F_HEADER_SIZE_MAX, mysink->output_buffer_bound).
> LZ4F_HEADER_SIZE_MAX is defined as 19 in lz4 library. With this
> proposal, it is almost guaranteed that the next buffer length will
> be always set to 19, which will result in failure of a call to
> LZ4F_compressUpdate() with the error LZ4F_ERROR_dstMaxSize_tooSmall,
> even if we had called bbsink_archive_contents() before.

Sorry, should have been Max(), not Min().

> You mean the way gzip allows us to use our own alloc and free functions
> by means of providing the function pointers for them. Unfortunately,
> no, LZ4 does not have that kind of provision. Maybe that makes a
> good proposal for LZ4 library ;-).
> I cannot think of another solution to it right away.

OK. Will give it some thought.

--
Robert Haas
EDB: http://www.enterprisedb.com