Re: [GENERAL] numeric precision when raising one numeric to another.

Bruce Momjian wrote:
> Alvaro Herrera wrote:
> > On Mon, May 30, 2005 at 11:29:48PM -0400, Bruce Momjian wrote:
> >
> > > test=> select 12345678901234567890 / 123;
> > > ?column?
> > > --------------------
> > > 100371373180768845
> > > (1 row)
> >
> > Well, that's a bug, right?
>
> I don't think so. The fuller answer is
> 100371373180768844.63414634146341463414, and that rounded to the nearest
> integer is 100371373180768845. I think people expect % do to that,
> except for integers. You could argue that numerics with zero scale are
> integers, but NUMERIC % NUMERIC doesn't behave like an integer operator
> --- it rounds to the proper precision.

Attached is a fix for the problem:

test=> select 12345678901234567890 % 123;
?column?
----------
78
(1 row)

I could have just forced X digits after the decimal point, but there was
really no _right_ number of digits, so it would have been a hack. The
proper solution is to change div_var to accept 'round' as true/false,
and that's what I did in the patch.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073