Re: Incorrect rounding of double values at max precision

Andrew Gierth <andrew(at)tao11(dot)riddles(dot)org(dot)uk> writes:
> "Tom" == Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> Tom> The other thing we could conceivably do is ask sprintf for more
> Tom> digits. But since those extra digit(s) aren't fully precise, I'm
> Tom> afraid that would likewise introduce as many oddities as it fixes.
> Tom> Still, it's somewhat interesting to wonder whether applying the
> Tom> Ryu algorithm would produce better or worse results on average.

> Hmm.

> The Ryu output values will still throw out edge cases similar to the
> above; for example, 502.15::float8 / 10 = 50.214999999999996 whereas
> 502.15::numeric / 10 = 50.215, so rounding the result of that to 2
> digits will give a different result.

Yeah. Worse, casting that to numeric currently gives the "correct"
result:

regression=# select (502.15::float8 / 10)::numeric;
numeric
---------
50.215
(1 row)

while if we changed float8_numeric to apply Ryu, the result would be
50.214999999999996. So that's not great. But there are other cases
where the result would be better than before, such as the OP's example
of 42258656681.38498::float8. I'd like to get my hands around how
many "better" and "worse" cases there would be, but I'm not sure how
to attack that question.

> Perhaps it would make more sense for the float8 to numeric cast to look
> at the requested typmod and use that for the conversion?

As things stand right now, float8_numeric has no idea what the target
typmod is; any typmod-driven rounding happens in a separate function
call afterwards. I don't recall whether the parser's casting
infrastructure could support merging those steps, and I'm not sure
it matters in most cases. Commonly, we don't have a target typmod.
(Still, if we do, having two separate rounding steps isn't nice.)

regards, tom lane