Re: pg_dump -Fd and compression level

On 07/25/2015 10:50 AM, Michael Paquier wrote:
> On Sat, Jul 25, 2015 at 11:09 PM, Marc Mamin <M(dot)Mamin(at)intershop(dot)de> wrote:
>>> On Sat, Jul 25, 2015 at 9:56 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>>>> On 07/25/2015 03:20 AM, Andrew Dunstan wrote:
>>>>>
>>>>> On 07/25/2015 02:34 AM, Marc Mamin wrote:
>>>>>>
>>>>>>> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>>>>>>>> Hmm. Yeah. It looks like commit
>>>>>>>> a7ad5cf0cfcfab8418000d652fa4f0c6ad6c8911
>>>>>>>> changed from using the default compression for libz to using the
>>>>>>>> compression set in pg_dump options, which defaults to 0. This actually
>>>>>>>> seems like the right thing to do, but it certainly should have been
>>>>>>>> called out much more forcefully in release notes, and arguably should
>>>>>>>> not have been changed in stable releases. Not sure what we do about it
>>>>>>>> now.
>>>>>> really 0? wouldn't that mean no compression at all?
>>>>>
>>>>> No, that's not right either. The default should be Z_DEFAULT_COMPRESSION,
>>>>> so we shouldn't actually see a difference in the default case. And it is
>>>>> definitely compressing some. So I'm now puzzled by what you're seeing.
>>>>>
>>>>>
>>>>
>>>> OK, I have got this worked out. I'll have a bug fix shortly.
>>> So you are basically planning to switch to Z_BEST_SPEED instead of
>>> Z_DEFAULT_COMPRESSION where needed for the default code path?
>> It looks like Z_DEFAULT_COMPRESSION was the used value before, and from the source code I guess this should still be the case.
>>
>> From a quick testing, it now behaves as if the minus sign from Z_DEFAULT_COMPRESSION is lost on the way, resulting in -Z1. this might indicate that Z_DEFAULT_COMPRESSION is parsed like the command line argument.

gzopen only allows a digit, so -1 results in the - being ignored and 1
being the compression value. It's a pity gzopen is so liberal about
accepting modes with values it ignores, or we'd have noticed this ages ago.

> I think I understand what is happening... With a quick test using the
> default compression level in directory mode, gzopen is called with
> w-1. By decrypting the docs of zlib (http://www.zlib.net/manual.html)
> section File Access Functions, it seems to me that calling gzopen like
> that will cause the file to not be compressed at all, which is
> actually why you are seeing an increase in your dump files, while we
> should call it with a compression mode of 6 actually based on what
> Z_DEFAULT_COMPRESSION means for zlib now, or simply without a
> compression level specified such as the default is used by zlib. Prior
> to a7ad5cf0, what we did was actually gzopen with always "w" or "wb"
> that caused the default compression level of 6 to be used. Hence I
> think that we should definitely use "w" or "wb" without specifying a
> level number when the compression level is Z_DEFAULT_COMPRESSION, and
> "wbN" (0<=N<=9) when a compression level is given by the user. A patch
> is attached for this purpose. Marc, does it work for you?
>
>> e.g. pg_dump -Z-2 is like pg_dump -Z2
> The compression needs to be included in range [0,9].

I propose to tighten pg_dump's rules so that only 0..9 are accepted as
arguments for -Z, and in compress_io.c:cfopen(), if compression is equal
to Z_DEFAULT_COMPRESSION, not add any explicit compression value to the
mode, thus using the zlib default.

cheers

andrew