Re: Using quicksort for every external sort run

On Wed, Dec 23, 2015 at 7:48 PM, Peter Geoghegan <pg(at)heroku(dot)com> wrote:
> I wonder, what's the situation here like with the attached patch
> applied on top of what you were testing? I think that we might be
> better off with more merge steps when under enormous memory pressure
> at the low end, in order to be able to store more tuples per tape (and
> do more sorting using quicksort).

Actually, now that I look into it, I think your 64MB work_mem setting
would have 234 tapes in total, so my patch won't do anything for your
case. Maybe change MAXORDER to 100 within the patch, to see where that
leaves things? I want to see if there is any improvement.

234 tapes means that approximately 5.7MB of memory would go to just
using tapes (for accounting purposes, which is mostly my concern
here). However, for a case like this, where you're well short of being
able to do everything in one pass, there is no benefit to having more
than about 6 tapes (I guess that's probably still true these days).
That 5.7MB of tape space for accounting purposes (and also in reality)
may not only increase the amount of random I/O required, and not only
throw off the memtuples estimate within grow_memtuples() (its balance
against everything else), but also decrease the cache efficiency in
the final on-the-fly merge (the efficiency in accessing tuples).

--
Peter Geoghegan