pgsql: Fix lossy KNN GiST when ordering operator returns non-float8 val

Fix lossy KNN GiST when ordering operator returns non-float8 value.

KNN GiST with recheck flag should return to executor the same type as ordering
operator, GiST detects this type by looking to return type of function which
implements ordering operator. But occasionally detecting code works after
replacing ordering operator function to distance support function.
Distance support function always returns float8, so, detecting code get float8
instead of actual return type of ordering operator.

Built-in opclasses don't have ordering operator which doesn't return
non-float8 value, so, tests are impossible here, at least now.

Backpatch to 9.5 where lozzy KNN was introduced.

Author: Alexander Korotkov
Report by: Artur Zakirov

Branch
------
REL9_5_STABLE

Details
-------
http://git.postgresql.org/pg/commitdiff/62e0ade9aa89c8f88ce506e1cb4be6e3697ee2d0

Modified Files
--------------
src/backend/access/gist/gistscan.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)