Re: Refactoring of compression options in pg_basebackup

On Mon, Dec 20, 2021 at 10:19:44AM -0500, Robert Haas wrote:
> One thing we should keep in mind is that I'm also working on adding
> server-side compression, initially with gzip, but Jeevan Ladhe has
> posted patches to extend that to LZ4. So however we structure the
> options they should take that into account. Currently that patch set
> adds --server-compression={none,gzip,gzipN} where N from 1 to 9, but
> perhaps it should be done differently. Not sure.

Yeah, consistency would be good. For the client-side compression of
LZ4, we have shaped things around the existing --compress option, and
there is 6f164e6 that offers an API to parse that at option-level,
meaning less custom error strings. I'd like to think that splitting
the compression level and the compression method is still the right
choice, except if --server-compression combined with a client-side
compression is a legal combination. This would not really make sense,
though, no? So we'd better block this possibility from the start?

>> One more thing that I have noticed while hacking this stuff is that we
>> have no regression tests for gzip with pg_basebackup, so I have added
>> some that are skipped when not compiling the code with ZLIB.
>
> If they don't decompress the backup and run pg_verifybackup on it then
> I'm not sure how much they help. Yet, I don't know how to do that
> portably.

They help in checking that an environment does not use a buggy set of
GZIP, at least. Using pg_verifybackup on a base backup with
--format='t' could be tweaked with $ENV{TAR} for the tarballs
generation, for example, as we do in some other tests. Option sets
like "xvf" or "zxvf" should be rather portable across the buildfarm,
no? I'd like to think that this is not a requirement for adding
checks in the compression path, as a first step, though, but I agree
that it could be extended more.
--
Michael