Re: Compression and on-disk sorting

On Wed, 2006-06-07 at 01:33 -0500, Jim C. Nasby wrote:
> On Fri, May 26, 2006 at 09:21:44PM +0100, Simon Riggs wrote:
> > On Fri, 2006-05-26 at 14:47 -0500, Jim C. Nasby wrote:
> >
> > > But the meat is:
> > > -- work_mem --
> > > Scale 2000 20000
> > > not compressed 150 805.7 797.7
> > > not compressed 3000 17820 17436
> > > compressed 150 371.4 400.1
> > > compressed 3000 8152 8537
> > > compressed, no headers 3000 7325 7876
> >
> > Since Tom has committed the header-removing patch, we need to test
> >
> > not compressed, no headers v compressed, no headers
>
> -- work_mem --
> Scale 2000 20000
> not compressed 150 805.7 797.7
> not compressed 3000 17820 17436
> not compressed, no hdr 3000 14470 14507
> compressed 150 371.4 400.1
> compressed 3000 8152 8537
> compressed, no headers 3000 7325 7876

That looks fairly conclusive. Can we try tests with data in reverse
order, so we use more tapes? We're still using a single tape, so the
additional overhead of compression doesn't cause any pain.

> > There is a noticeable rise in sort time with increasing work_mem, but
> > that needs to be offset from the benefit that in-general comes from
> > using a large Heap for the sort. With the data you're using that always
> > looks like a loss, but that isn't true with all input data orderings.
>
> I thought that a change had been made to the on-disk sort specifically to
> eliminate the problem of more work_mem making the sort take longer.

There was a severe non-optimal piece of code...but the general effect
still exists. As does the effect that having higher work_mem produces
fewer runs which speeds up the final stages of the sort.

> I also
> thought that there was something about that fix that was tunable.

Increasing work_mem makes *this* test case take longer.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com