Re: Incorrect rounding of double values at max precision

Andres Freund <andres(at)anarazel(dot)de> writes:
> On 2020-10-20 21:48:52 -0400, Tom Lane wrote:
>> Meh. Yeah, we could improve the observed results for float4 values
>> that are integers between 1M and 16M, and some similarly-sized band
>> for float8; but to what end?

> I'm not actually arguing that we should improve it by relying on range
> based heuristics. Just that throwing away precision that can't just
> argued to have been conjured by float representation issues is an
> indicator of our current approach to be quite wrong.

I think your argument is founded on the unprovable assumption that
the original input value was an integer. If we see 12345678::float4,
the "true" value could have been anything between 12345677.5 and
12345678.5. Printing it as an exact integer isn't going to seem
too exact to a person who knows they put in a fraction.

Or for a person who does know a little bit about float arithmetic, they
might reasonably wonder why 12345.678::float4 is "exactly" converted to
numeric but 1234.5678::float4, with the same number of digits, is not.
(Ryu renders the latter as 1234.5677.)

>> The most likely practical result is just to postpone the user's
>> discovery that they're Doing It Wrong.

> I don't buy this, not even for a second. Why is FLT_DIG = 6 the right
> way to hint at that? Why not a precision of 5, 4, 3? Sure 6 digits is
> guaranteed to roundtrip the same way, but that's an *extremely* coarse
> filter.

It's also the *highest* precision that is guaranteed to round-trip,
which 5,4,3 are not, so that argument is unconvincing too.

> And if we like exposing people to floating point imprecision, why is
> select (502.15::float8 / 10)::numeric resulting in 50.214999999999996
> bad?

The point there was that that's "worse" (seemingly less precise)
than what happens now.

Please notice that I did not say that that one example means we
shouldn't change; any more than I think that isolated other examples
mean we should. The question I was trying to get at is how
we could decide whether using Ryu for this is an overall win when
some examples will get better and others worse. No one's stepped
up to that plate yet.

regards, tom lane