Re: Optimize mul_var() for var1ndigits >= 8

On Tue, 13 Aug 2024 at 08:49, Joel Jacobson <joel(at)compiler(dot)org> wrote:
>
> I reran the tests and v5 produces much fewer diffs than v4.
> Not sure if the remaining ones are problematic or not.
>
> joel(at)Joels-MBP postgresql % ./test-mul_var-verify-v5.sh
> HEAD is now at a67a49648d Rename C23 keyword
> SET
> DROP TABLE
> CREATE TABLE
> COPY 14130000
> Expanded display is on.
> -[ RECORD 1 ]------+-------------------------------
> var1 | 0.0000000000000000489673392928
> var2 | 6.713030439846337
> rscale_adjustment | -15
> expected | 0.0000000000000003287192392308
> numeric_mul_rscale | 0.0000000000000003287192392309
> diff | 0.0000000000000000000000000001
>

Yes, that's exactly the sort of thing you'd expect to see. The exact
product of var1 and var2 in that case is

0.0000_0000_0000_0003_2871_9239_2308_5000_4574_2504_736

so numeric_mul_rscale() with the patch is producing the correctly
rounded result, and "expected" is the result from HEAD, which is off
by 1 in the final digit.

To make it easier to hit such cases, I tested with the attached test
script, which intentionally produces pairs of numbers whose product
contains '5' followed by 5 zeros, and rounds at the digit before the
'5', so the correct answer should round up, but the truncated product
is quite likely not to do so.

With HEAD, this gives 710,017 out of 1,000,000 cases that are off by 1
in the final digit (always 1 too low in the final digit), and with the
v5 patch, it gives 282,595 cases. Furthermore, it's an exact subset:

select count(*) from diffs1; -- HEAD
count
--------
710017
(1 row)

pgdevel=# select count(*) from diffs2; -- v5 patch
count
--------
282595
(1 row)

select * from diffs2 except select * from diffs1;
n | z | m | w | x | y | expected | numeric_mul_rscale
---+---+---+---+---+---+----------+--------------------
(0 rows)

which is exactly what I was hoping to see (no cases where the patch
made it less accurate).

Regards,
Dean