Re: PoC: Duplicate Tuple Elidation during External Sort for DISTINCT

Jon Nelson <jnelson+pgsql(at)jamponi(dot)net> writes:
> A rough summary of the patch follows:

> - a GUC variable enables or disables this capability
> - in nodeAgg.c, eliding duplicate tuples is enabled if the number of
> distinct columns is equal to the number of sort columns (and both are
> greater than zero).
> - in createplan.c, eliding duplicate tuples is enabled if we are
> creating a unique plan which involves sorting first
> - ditto planner.c
> - all of the remaining changes are in tuplesort.c, which consist of:
> + a new macro, DISCARDTUP and a new structure member, discardtup, are
> both defined and operate similar to COMPARETUP, COPYTUP, etc...
> + in puttuple_common, when state is TSS_BUILDRUNS, we *may* simply
> throw out the new tuple if it compares as identical to the tuple at
> the top of the heap. Since we're already performing this comparison,
> this is essentially free.
> + in mergeonerun, we may discard a tuple if it compares as identical
> to the *last written tuple*. This is a comparison that did not take
> place before, so it's not free, but it saves a write I/O.
> + We perform the same logic in dumptuples

[ raised eyebrow ... ] And what happens if the planner drops the
unique step and then the sort doesn't actually go to disk?

regards, tom lane