Re: How can I accurately infer the output of a SELECT statement FROM a table with floating-point data type?

On 2023-10-19 00:12:51 -0400, Tom Lane wrote:
> =?GBK?B?w6vDqw==?= <krave(at)163(dot)com> writes:
> > Do we have a rule by follow which one can accurately info the output of a SELECT statment FROM a table with floating-point data type?
>
> The goal of our floating-point output functions is to produce
> the shortest representation from which the stored value could
> be recovered exactly. It's not unusual for that to require
> two or three decimal digits more than the nominal precision.

To illustrate this:

A real has 24 bits of mantissa. 12345.123456789 is between 2**13 and
2**14, so there are 10 bits left for the fraction. Or in other words,
the number must be approximated as a multiple of 1/1024.

The closest we can get is 12345+124/1024 = 12345.123046875

12345.123 would obviously be rounded to the same number, so it's close
enough and additional digits aren't necessary.

But 12345.12 would be rounded to 12345+123/1024 = 12345.1201171875.
That's different, so 7 digits are not enough in this case.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp(at)hjp(dot)at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"