From: | Theo Kramer <theo(at)flame(dot)flame(dot)co(dot)za> |
---|---|
To: | pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: [HACKERS] 123.45 - 123 = 0.45 |
Date: | 1998-12-16 05:22:41 |
Message-ID: | 199812160522.HAA06771@flame.flame.co.za |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
> We need to define what the precision of a result should be,
> if it is not assigned to a column (where the precision can be
> the atttypmod). Is there any standard defined for? If not,
> what about this:
>
> Internal representation holds different precision for DISPLAY
> and CALC.
>
> On any operation, the DISPLAY precision is set to the higher
> of the two operands.
>
> On add/subtract, the CALC precision becomes the higher of the
> two.
>
> On multiply, the CALC precision is adjusted to hold the exact
> result up to a (variable settable?) maximum.
>
> On divide, the CALC precision is set to max and after it to
> the number of used digits.
>
> If the result get's assigned to an attribute, it is rounded
> to it's atttypmod and both precisions set to that.
>
> The types output function rounds it to the DISPLAY precision.
>
> The input function sets both precisions to the number of
> digits present after decimal point.
>
> Needless to say that there will be special functions to round
> explicitly and set the precisions.
I have a routine that does the necessary rounding on 8 byte floating points to a
precision up to 8 decimal places. Not exactly based on higher math but it
does the job in many financial applications on all assignments, comparisons
and when displayed.
I never use it on arithmetic operations.
The interface is as follows
double RoundDouble(double value, int decimals);
I am happy to submit it for use in the postgres code.
Regards
Theo
From | Date | Subject | |
---|---|---|---|
Next Message | Vadim Mikheev | 1998-12-16 11:00:56 | CURRENT: crash in select_view regression test... |
Previous Message | dustin sallings | 1998-12-16 03:00:28 | Re: [GENERAL] Row Level Locking, On-line Recovery |