BUG #6399: knngist sometimes returns tuples in incorrect order

The following bug has been logged on the website:

Bug reference: 6399
Logged by: YAMAMOTO Takashi
Email address: yamt(at)mwd(dot)biglobe(dot)ne(dot)jp
PostgreSQL version: Unsupported/Unknown
Operating system: NetBSD
Description:

9.2devel
(01d83ffdcae92f75dbfd41de0b4213d241edd394)

knngist seems to assume that any distances can be represented with float8.
at least distances between int8 values can not.

the following example uses btree_gist extension.
results should be the same regardless of the existance of the index.

create temp table t (a int8);
insert into t values (9223372036854775806),(9223372036854775807);
select *,0<->a as dist from t order by dist;
create index on t using gist (a);
set enable_seqscan=off;
select *,0<->a as dist from t order by dist;

CREATE TABLE
INSERT 0 2
a | dist
---------------------+---------------------
9223372036854775806 | 9223372036854775806
9223372036854775807 | 9223372036854775807
(2 rows)

CREATE INDEX
SET
a | dist
---------------------+---------------------
9223372036854775807 | 9223372036854775807
9223372036854775806 | 9223372036854775806
(2 rows)