Re: Compression and on-disk sorting

On Mon, May 15, 2006 at 02:18:03PM -0400, Tom Lane wrote:
> "Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
> > A recent post Tom made in -bugs about how bad performance would be if we
> > spilled after-commit triggers to disk got me thinking... There are
> > several operations the database performs that potentially spill to disk.
> > Given that any time that happens we end up caring much less about CPU
> > usage and much more about disk IO, for any of these cases that use
> > non-random access, compressing the data before sending it to disk would
> > potentially be a sizeable win.
>
> Note however that what the code thinks is a spill to disk and what
> actually involves disk I/O are two different things. If you think
> of it as a spill to kernel disk cache then the attraction is a lot
> weaker...

I'm really starting to see why other databases want the OS out of their
way...

I guess at this point the best we could do would be to have a
configurable limit for when compression started. The first X number of
bytes go out uncompressed, everything after that is compressed. I don't
know of any technical reason why you couldn't switch in the middle of a
file, so long as you knew exactly where you switched.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461