Re: PoC: Duplicate Tuple Elidation during External Sort for DISTINCT

On 22/01/14 03:53, Tom Lane wrote:
> Jon Nelson <jnelson+pgsql(at)jamponi(dot)net> writes:
>> A rough summary of the patch follows:
>
>> - a GUC variable enables or disables this capability
>> - in nodeAgg.c, eliding duplicate tuples is enabled if the number of
>> distinct columns is equal to the number of sort columns (and both are
>> greater than zero).
>> - in createplan.c, eliding duplicate tuples is enabled if we are
>> creating a unique plan which involves sorting first
>> - ditto planner.c
>> - all of the remaining changes are in tuplesort.c, which consist of:
>> + a new macro, DISCARDTUP and a new structure member, discardtup, are
>> both defined and operate similar to COMPARETUP, COPYTUP, etc...
>> + in puttuple_common, when state is TSS_BUILDRUNS, we *may* simply
>> throw out the new tuple if it compares as identical to the tuple at
>> the top of the heap. Since we're already performing this comparison,
>> this is essentially free.
>> + in mergeonerun, we may discard a tuple if it compares as identical
>> to the *last written tuple*. This is a comparison that did not take
>> place before, so it's not free, but it saves a write I/O.
>> + We perform the same logic in dumptuples
>
> [ raised eyebrow ... ] And what happens if the planner drops the
> unique step and then the sort doesn't actually go to disk?

I don't think Jon was suggesting that the planner drop the unique step.
--
Cheers,
Jeremy