Re: rounding problems

Sam Mason wrote:
> On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
>
>> Sam Mason wrote:
>>
>>> What does foxpro use for storing numbers? or is it just that you never
>>> pushed it hard enough for the abstractions to show through.
>>>
>> I know i pushed it. Foxpro for the most has only 4 basic data types
>> Numeric (similar to Posgresql numeric), Boolean, Date, Text aka
>> (string) The foxpro tables supported far more data types but when every
>> it was dumped to variable it acted like one of the 4.
>>
>
> I really meant how much did you check the results, or did you accept
> that they were correct?
>
>
>> Foxpro did not suffer floating point math errors. I normally used 8 to
>> 10 points precision. Foxpro was limited to 15 points of precision
>> period. No more and no less, once you hit that was it.
>>
>
> 15 places seems very similar to what a 64bit IEEE floating point number
> will give you, i.e. a double in C/C++.
>
>
>> My problem is we calculate resistance of parts in a Foxpro app that we
>> want to move because we want to bring all the custom apps into one
>> framework and single database.
>>
>> Take this calculation (0.05/30000* 1.0025) which is used to calculate
>> parts resistance and Tolerance. (its Ohms Law) The value returned from
>> C++ = .0000016708 which is wrong
>> it should be .00000167418. We just shrank the tolerance on the part we
>> make
>>
>
> Why are you so sure about the FoxPro result? I've just checked a few
> calculators and get results consistent with your C++ version.
>
> Justin C: 0.0000016708
> J FoxPro: 0.00000167418
>
this 167418 came of my ti 89 calculator, going back i noticed that i
accident rounded it to .00000167 which gives a bad result.

So what i typed in after that point is wrong. OOPS.

But loosing the 3 will put out of the tolerance sense its the last
significant digit needed thats displayed on the measurement devices. So
if the 3 becomes a 4 your out of tolerance.
> My C: 0.000001670833
> bc[1]: 0.0000016708333333333333333333333333333332
> PG[2]: 0.0000016708333333333333336675
>
> Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
>
Foxpro Agrees with what you have 0.00000167083333
the code looks like this

SET DECIMALS TO 15
? ((0.05/30000)* 1.0025)

When i wrote the application like 10 years ago I spent allot time making
sure the numbers where correct even doing some by hand.

If I gotten it wrong there's allot National labs, Universities, Big
companies that are generating allot bad results in their QC departments.

Chced
> Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
> the math, and as they all agree I'm thinking FoxPro is incorrect!

Here is the foxpro Documentation

Integers or decimal numbers

For example, the quantity of items ordered

8 bytes in memory; 1 to 20 bytes in table

- .9999999999E+19 to .9999999999E+20

> Next
> I tried doing it accurately (in Haskell if it makes any difference) and
> get an answer of 401/240000000 out, which would agree with everything
> but FoxPro. If I calculate the ratio back out for FoxPro I get
> 401/239520242 which is a little way out.
>
>
>> The Documentation from MS says 15 points of precision but the result say
>> otherwise.
>>
>
> The docs for what? FoxPro or their C compiler?
>
From the MS Document here is Copied text

*Microsoft Specific --->*

The double type contains 64 bits: 1 for sign, 11 for the exponent, and
52 for the mantissa. Its range is +/--1.7E308 with at least 15 digits of
precision.

*END Microsoft Specific*

> If you mean FoxPro, I think this is another case of MS screwing up.
>
Foxpro normally did not suffer form other MS screw ups.
>
>> I'm glad You and others are taking the time to explain to me
>> the odd results before i get into redoing that application.
>>
>
> Welcome to the PG community, lots of people to get interested in lots of
> things!
>
>
>> Why oh Why did MS kill Foxpro. :'( I understood it, knew its quirks
>> and it worked very well with Postgresql
>>
>
> Are you sure you want to stay with it if its answers are wrong?
>
>
> Sam
>
> [1] http://www.gnu.org/software/bc/manual/html_mono/bc.html
> [2] http://doxygen.postgresql.org/backend_2utils_2adt_2numeric_8c-source.html
> [3] http://www.google.com/search?q=0.05/30000*1.0025
>
>