Re: [HACKERS] Custom compression methods

On Sun, Feb 28, 2021 at 9:48 AM Justin Pryzby <pryzby(at)telsasoft(dot)com> wrote:
>
> On my PC, this new change is causing a test failure:
>
> SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
> - substr
> -----------------------------------------------------
> - 01234567890123456789012345678901234567890123456789
> -(1 row)
> -
> +ERROR: compressed lz4 data is corrupt

The older version of lz4 had this problem that while decompressing
partial if we don't give the buffer size up to full data length it was
failing[1] but it is solved in version 1.9.

> @@ -119,15 +119,15 @@ lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength)
> int32 rawsize;
> struct varlena *result;
>
> - /* allocate memory for holding the uncompressed data */
> - result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ);
> + /* allocate memory for the uncompressed data */
> + result = (struct varlena *) palloc(slicelength + VARHDRSZ);
>
> - /* decompress partial data using lz4 routine */
> + /* decompress the data */
> rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS,
> VARDATA(result),
> VARSIZE(value) - VARHDRSZ_COMPRESS,
> slicelength,
> - VARRAWSIZE_4B_C(value));
> + slicelength);

This is done for the latest version as now we don't need to allocate
the buffer of full size, it is enough the allocate just equal to the
slicelength.

> Also, in the tests, you have this at both the top and bottom of the file:
>
> src/test/regress/sql/compression.sql:\set HIDE_COMPRESSAM false
> src/test/regress/sql/compression.sql:\set HIDE_COMPRESSAM false
>
> Whereas the patch I sent had at the end:
>
> +\set HIDE_COMPRESSAM on
>
> ("on" is the default when run under pg_regress)

I will fix this.

[1] https://docs.unrealengine.com/en-US/API/Runtime/Core/Compression/LZ4_decompress_safe_partial/index.html

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com