| From: | Justin Pryzby <pryzby(at)telsasoft(dot)com> |
|---|---|
| To: | Dipesh Pandit <dipesh(dot)pandit(at)gmail(dot)com> |
| Cc: | Robert Haas <robertmhaas(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, Jeevan Ladhe <jeevanladhe(dot)os(at)gmail(dot)com>, Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>, "Shinoda, Noriyoshi (PN Japan FSIP)" <noriyoshi(dot)shinoda(at)hpe(dot)com>, Abhijit Menon-Sen <ams(at)toroid(dot)org>, Dmitry Dolgov <9erthalion6(at)gmail(dot)com>, Jeevan Ladhe <jeevan(dot)ladhe(at)enterprisedb(dot)com>, Mark Dilger <mark(dot)dilger(at)enterprisedb(dot)com>, pgsql-hackers(at)postgresql(dot)org, tushar <tushar(dot)ahuja(at)enterprisedb(dot)com> |
| Subject: | Re: refactoring basebackup.c (zstd workers) |
| Date: | 2022-03-14 16:35:47 |
| Message-ID: | 20220314163547.GT28503@telsasoft.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
On Mon, Mar 14, 2022 at 09:41:35PM +0530, Dipesh Pandit wrote:
> I tried to implement support for parallel ZSTD compression. The
> library provides an option (ZSTD_c_nbWorkers) to specify the
> number of compression workers. The number of parallel
> workers can be set as part of compression parameter and if this
> option is specified then the library performs parallel compression
> based on the specified number of workers.
>
> User can specify the number of parallel worker as part of
> --compress option by appending an integer value after at sign (@).
> (-Z, --compress=[{client|server}-]{gzip|lz4|zstd}[:LEVEL][(at)WORKERS])
I suggest to use a syntax that's more general than that, maybe something like
:[level=]N,parallel=N,flag,flag,...
For example, someone may want to use zstd "long" mode or (when it's released)
rsyncable mode, or specify fine-grained compression parameters (strategy,
windowLog, hashLog, etc).
I hope the same syntax will be shared with wal_compression and pg_dump.
And libpq, if that patch progresses.
BTW, I think this may be better left for PG16.
--
Justin
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Dilip Kumar | 2022-03-14 16:44:27 | Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints |
| Previous Message | Robert Haas | 2022-03-14 16:34:04 | Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints |