Incorrect results from numeric round() and trunc()

From: Dean Rasheed <dean(dot)a(dot)rasheed(at)gmail(dot)com>
To: PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Incorrect results from numeric round() and trunc()
Date: 2024-07-07 11:28:32
Message-ID: CAEZATCXB+rDTuMjhK5ZxcouufigSc-X4tGJCBTMpZ3n=xxQuhg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

The numeric round() and trunc() functions clamp the scale argument to
the range between +/- NUMERIC_MAX_RESULT_SCALE, which is +/- 2000.
That's a long way short of the actual allowed range of type numeric,
so they produce incorrect results when rounding/truncating more than
2000 digits before or after the decimal point. For example,
round(1e-5000, 5000) returns 0 instead of 1e-5000.

Attached is a patch fixing that, using the actual documented range of
type numeric.

I've also tidied up a bit by replacing all instances of SHRT_MAX with
a new constant NUMERIC_WEIGHT_MAX, whose name more accurately
describes the limit, as used in various other overflow checks.

In doing so, I also noticed a comment in power_var() which claimed
that ln_dweight could be as low as -SHRT_MAX (-32767), which is wrong.
It can only be as small as -NUMERIC_DSCALE_MAX (-16383), though that
doesn't affect the point being made in that comment.

I'd like to treat this as a bug-fix and back-patch it, since the
current behaviour is clearly broken.

Regards,
Dean

Attachment Content-Type Size
fix-numeric-round-and-trunc-limits.patch text/x-patch 8.3 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Egor Rogov 2024-07-07 12:21:51 Re: Multi-transactional statements and statistics for autovacuum
Previous Message Andrew Dunstan 2024-07-07 11:28:28 Re: 010_pg_basebackup.pl vs multiple filesystems