Re: pg_dump -Fd and compression level

On Sat, Jul 25, 2015 at 11:09 PM, Marc Mamin <M(dot)Mamin(at)intershop(dot)de> wrote:
>
>>On Sat, Jul 25, 2015 at 9:56 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>>>
>>> On 07/25/2015 03:20 AM, Andrew Dunstan wrote:
>>>>
>>>>
>>>> On 07/25/2015 02:34 AM, Marc Mamin wrote:
>>>>>
>>>>>
>>>>>> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>>>>>>>
>>>>>>> Hmm. Yeah. It looks like commit
>>>>>>> a7ad5cf0cfcfab8418000d652fa4f0c6ad6c8911
>>>>>>> changed from using the default compression for libz to using the
>>>>>>> compression set in pg_dump options, which defaults to 0. This actually
>>>>>>> seems like the right thing to do, but it certainly should have been
>>>>>>> called out much more forcefully in release notes, and arguably should
>>>>>>> not have been changed in stable releases. Not sure what we do about it
>>>>>>> now.
>>>>>
>>>>> really 0? wouldn't that mean no compression at all?
>>>>
>>>>
>>>> No, that's not right either. The default should be Z_DEFAULT_COMPRESSION,
>>>> so we shouldn't actually see a difference in the default case. And it is
>>>> definitely compressing some. So I'm now puzzled by what you're seeing.
>>>>
>>>>
>>>
>>>
>>> OK, I have got this worked out. I'll have a bug fix shortly.
>>
>>So you are basically planning to switch to Z_BEST_SPEED instead of
>>Z_DEFAULT_COMPRESSION where needed for the default code path?
>
> It looks like Z_DEFAULT_COMPRESSION was the used value before, and from the source code I guess this should still be the case.
>
> From a quick testing, it now behaves as if the minus sign from Z_DEFAULT_COMPRESSION is lost on the way, resulting in -Z1. this might indicate that Z_DEFAULT_COMPRESSION is parsed like the command line argument.

I think I understand what is happening... With a quick test using the
default compression level in directory mode, gzopen is called with
w-1. By decrypting the docs of zlib (http://www.zlib.net/manual.html)
section File Access Functions, it seems to me that calling gzopen like
that will cause the file to not be compressed at all, which is
actually why you are seeing an increase in your dump files, while we
should call it with a compression mode of 6 actually based on what
Z_DEFAULT_COMPRESSION means for zlib now, or simply without a
compression level specified such as the default is used by zlib. Prior
to a7ad5cf0, what we did was actually gzopen with always "w" or "wb"
that caused the default compression level of 6 to be used. Hence I
think that we should definitely use "w" or "wb" without specifying a
level number when the compression level is Z_DEFAULT_COMPRESSION, and
"wbN" (0<=N<=9) when a compression level is given by the user. A patch
is attached for this purpose. Marc, does it work for you?

> e.g. pg_dump -Z-2 is like pg_dump -Z2

The compression needs to be included in range [0,9].
--
Michael