Re: TOAST compression

Hannu,

On 2/26/06 12:19 PM, "Hannu Krosing" <hannu(at)skype(dot)net> wrote:

>> On DBT-3 data, I've just run some tests meant to simulate the speed
>> differences of compression versus native I/O. My thought is that an
>> external use of gzip on a binary dump file should be close to the speed of
>> LZW on toasted fields,
>
> Your basic assumption si probbaly wrong :(
>
> gzip what ? "compression level" setting of gzip has big effect on both
> compression speed and compression rate. And I suspect that even the
> fastest level (gzip -1) compresses slower and better than postgresql's
> lzcompress.

I thought it might form a lower bound on speed. 75MB/s on an Opteron is
really super fast compared to what I expected from LZ. And since gzip uses
LZ compression, maybe they'll be in the same family of results,
behavior-wise.

I'd be more concerned about the implementation within the executor and how
the "get tuple, decompress tuple, use tuple" cycling might destroy the
efficiency of the scanning compared to decompressing large blocks of data.

As Jim pointed out, we would need a real test to confirm the behavior, I'm
not yet acquainted with the toast compression, so it's harder for me to
compose a real test.

> these are also somewhat bogus tests, if you would want them to be
> comparable with dd below, you should have used 'time gzip -c
> supplier.bin > /dev/null'

Hmm - the file writing seems to matter on decompress, making decompression
even faster:

[root(at)salerno0 tmp]# time gunzip tmp.bin.gz

real 0m2.254s
user 0m1.718s
sys 0m0.536s
[root(at)salerno0 tmp]# time gzip tmp.bin

real 0m12.912s
user 0m12.555s
sys 0m0.355s
[root(at)salerno0 tmp]# time gzip -c supplier.bin > /dev/null

real 0m12.582s
user 0m12.464s
sys 0m0.115s
[root(at)salerno0 tmp]# time gunzip -c tmp.bin.gz > /dev/null

real 0m1.734s
user 0m1.690s
sys 0m0.043s

- Luke