From: | Justin Pryzby <pryzby(at)telsasoft(dot)com> |
---|---|
To: | Robert Haas <robertmhaas(at)gmail(dot)com> |
Cc: | Dipesh Pandit <dipesh(dot)pandit(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: multithreaded zstd backup compression for client and server |
Date: | 2022-03-23 21:52:34 |
Message-ID: | 20220323215234.GX28503@telsasoft.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
+ * We check for failure here because (1) older versions of the library
+ * do not support ZSTD_c_nbWorkers and (2) the library might want to
+ * reject an unreasonable values (though in practice it does not seem to do
+ * so).
+ */
+ ret = ZSTD_CCtx_setParameter(mysink->cctx, ZSTD_c_nbWorkers,
+ mysink->workers);
+ if (ZSTD_isError(ret))
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not set compression worker count to %d: %s",
+ mysink->workers, ZSTD_getErrorName(ret)));
Also because the library may not be compiled with threading. A few days ago, I
tried to rebase the original "parallel workers" patch over the COMPRESS DETAIL
patch but then couldn't test it, even after trying various versions of the zstd
package and trying to compile it locally. I'll try again soon...
I think you should also test the return value when setting the compress level.
Not only because it's generally a good idea, but also because I suggested to
support negative compression levels. Which weren't allowed before v1.3.4, and
then the range is only defined since 1.3.6 (ZSTD_minCLevel). At some point,
the range may have been -7..22 but now it's -131072..22.
lib/compress/zstd_compress.c:int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; }
lib/zstd.h:#define ZSTD_TARGETLENGTH_MAX ZSTD_BLOCKSIZE_MAX
lib/zstd.h:#define ZSTD_BLOCKSIZE_MAX (1<<ZSTD_BLOCKSIZELOG_MAX)
lib/zstd.h:#define ZSTD_BLOCKSIZELOG_MAX 17
; -1<<17
-131072
Attachment | Content-Type | Size |
---|---|---|
0001-pg_basebackup-support-Zstd-negative-compression-leve.txt | text/x-diff | 3.3 KB |
From | Date | Subject | |
---|---|---|---|
Next Message | Dmitry Dolgov | 2022-03-23 22:22:32 | Re: MDAM techniques and Index Skip Scan patch |
Previous Message | Tom Lane | 2022-03-23 21:32:46 | Re: MDAM techniques and Index Skip Scan patch |