Re: AdvanceXLInsertBuffer vs. WAL segment compressibility

(I'm cleaning up my inbox, hence the delayed reply)

On 08/02/2016 10:51 PM, Robert Haas wrote:
> On Tue, Aug 2, 2016 at 2:33 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>> On Tue, Jul 26, 2016 at 05:42:43PM -0400, Chapman Flack wrote:
>>> Even so, I'd be curious whether it would break anything to have
>>> xlp_pageaddr simply set to InvalidXLogRecPtr in the dummy zero
>>> pages written to fill out a segment. At least until it's felt
>>> that archive_timeout has been so decidedly obsoleted by streaming
>>> replication that it is removed, and the log-tail zeroing code
>>> with it.
>>>
>>> That at least would eliminate the risk of anyone else repeating
>>> my astonishment. :) I had read that 9.4 added built-in log-zeroing
>>> code, and my first reaction was "cool! that may make the compression
>>> technique we're using unnecessary, but certainly can't make it worse"
>>> only to discover that it did, by ~ 300x, becoming now 3x *worse* than
>>> plain gzip, which itself is ~ 100x worse than what we had.
>>
>> My guess is that the bytes are there to detect problems where a 512-byte
>> disk sector is zeroed by a disk failure. I don't see use changing that
>> for the use-case you have described.
>
> Is there actually any code that makes such a check?
>
> I'm inclined to doubt that was the motivation, though admittedly we're
> both speculating about the contents of Heikki's brain, a tricky
> proposition on a good day.

Given that we used to just leave them as garbage, it seems pretty safe
to zero them out now.

It's kind of nice that all the XLOG pages have valid page headers. One
way to think of the WAL switch record is that it's a very large WAL
record that just happens to consume the rest of the WAL segment. Except
that it's not actually represented like that; the xl_tot_len field of an
XLOG switch record does not include the zeroed out portion. Instead,
there's special handling in the reader code, that skips to the end of
the segment when it sees a switch record. So that point is moot.

When I wrote that code, I don't remember if I realized that we're
initializing the page headers, or if I thought that it's good enough
even if we do. I guess I didn't realize it, because a comment would've
been in order if it was intentional.

So +1 on fixing that, a patch would be welcome. In the meanwhile, have
you tried using a different compression program? Something else than
gzip might do a better job at the almost zero pages.

- Heikki