Re: rounding problems

Sam Mason wrote:
> On Wed, May 14, 2008 at 11:47:52AM -0400, Justin wrote:
>
>> I have forgotten how much i hate C++
>>
>
> What we're talking about doesn't have much to do with C++, it's floating
> point maths in general.
>
>
>> Its not doing what you say it would but it did do other odd ball
>> things. I miss my foxpro :-(.
>>
>
> What does foxpro use for storing numbers? or is it just that you never
> pushed it hard enough for the abstractions to show through.
>
I know i pushed it. Foxpro for the most has only 4 basic data types
Numeric (similar to Posgresql numeric), Boolean, Date, Text aka
(string) The foxpro tables supported far more data types but when every
it was dumped to variable it acted like one of the 4.

Foxpro did not suffer floating point math errors. I normally used 8 to
10 points precision. Foxpro was limited to 15 points of precision
period. No more and no less, once you hit that was it.

>
>> Plus its not holding 15 precision points
>>
>
> after changing the output to be:
>
> printf("%.10f %.10f\n", d, d-c);
>
> I get:
>
> 100000000.0999999940 0.0999999940
> 100000000.0100000054 0.0100000054
> 100000000.0010000020 0.0010000020
> 100000000.0001000017 0.0001000017
> 100000000.0000099987 0.0000099987
> 100000000.0000009984 0.0000009984
> 100000000.0000001043 0.0000001043
> 100000000.0000000149 0.0000000149
> 100000000.0000000000 0.0000000000
>
> Which looks reasonable. Remember that floating point numbers store
> their state in base two, not base ten. All of those numbers look good
> to 15 decimal digits.
>

My problem is we calculate resistance of parts in a Foxpro app that we
want to move because we want to bring all the custom apps into one
framework and single database.

Take this calculation (0.05/30000* 1.0025) which is used to calculate
parts resistance and Tolerance. (its Ohms Law) The value returned from
C++ = .0000016708 which is wrong
it should be .00000167418. We just shrank the tolerance on the part we
make

Take the other side (0.05/30000* .9975) = .0000016625 from C++ this
way wrong and the tolerance just grew .00000166583. Guess what this
could result in shipping a $12,000+ out to a customer wrong.

The Documentation from MS says 15 points of precision but the result say
otherwise. I'm glad You and others are taking the time to explain to me
the odd results before i get into redoing that application.

Why oh Why did MS kill Foxpro. :'( I understood it, knew its quirks
and it worked very well with Postgresql