From: | Dean Rasheed <dean(dot)a(dot)rasheed(at)gmail(dot)com> |
---|---|
To: | PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Incorrect results from numeric round() and trunc() |
Date: | 2024-07-07 11:28:32 |
Message-ID: | CAEZATCXB+rDTuMjhK5ZxcouufigSc-X4tGJCBTMpZ3n=xxQuhg@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
The numeric round() and trunc() functions clamp the scale argument to
the range between +/- NUMERIC_MAX_RESULT_SCALE, which is +/- 2000.
That's a long way short of the actual allowed range of type numeric,
so they produce incorrect results when rounding/truncating more than
2000 digits before or after the decimal point. For example,
round(1e-5000, 5000) returns 0 instead of 1e-5000.
Attached is a patch fixing that, using the actual documented range of
type numeric.
I've also tidied up a bit by replacing all instances of SHRT_MAX with
a new constant NUMERIC_WEIGHT_MAX, whose name more accurately
describes the limit, as used in various other overflow checks.
In doing so, I also noticed a comment in power_var() which claimed
that ln_dweight could be as low as -SHRT_MAX (-32767), which is wrong.
It can only be as small as -NUMERIC_DSCALE_MAX (-16383), though that
doesn't affect the point being made in that comment.
I'd like to treat this as a bug-fix and back-patch it, since the
current behaviour is clearly broken.
Regards,
Dean
Attachment | Content-Type | Size |
---|---|---|
fix-numeric-round-and-trunc-limits.patch | text/x-patch | 8.3 KB |
From | Date | Subject | |
---|---|---|---|
Next Message | Egor Rogov | 2024-07-07 12:21:51 | Re: Multi-transactional statements and statistics for autovacuum |
Previous Message | Andrew Dunstan | 2024-07-07 11:28:28 | Re: 010_pg_basebackup.pl vs multiple filesystems |