Re: rounding problems

> On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
>> My problem is we calculate resistance of parts in a Foxpro app
>> that we
>> want to move because we want to bring all the custom apps into one
>> framework and single database.
>>
>> Take this calculation (0.05/30000* 1.0025) which is used to
>> calculate
>> parts resistance and Tolerance. (its Ohms Law) The value
>> returned from
>> C++ = .0000016708 which is wrong it should be .00000167418.
>
> Why are you so sure about the FoxPro result? I've just checked a few
> calculators and get results consistent with your C++ version.
>
> Justin C: 0.0000016708
> J FoxPro: 0.00000167418
> My C: 0.000001670833
> bc[1]: 0.0000016708333333333333333333333333333332
> PG[2]: 0.0000016708333333333333336675
> Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
>
> Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
> the math, and as they all agree I'm thinking FoxPro is incorrect!
> Next
> I tried doing it accurately (in Haskell if it makes any difference)
> and
> get an answer of 401/240000000 out, which would agree with everything
> but FoxPro. If I calculate the ratio back out for FoxPro I get
> 401/239520242 which is a little way out.

I'll add my Casio scientific calculator to the list, which also gives
the non-FoxPro result.

We can also write this "exactly" as:

(0.05 * 1.0025 / 3) * 10^-5
= (5 * 10025 / 3) * 10^-11
= (50,125 / 3) * 0.00000000001
= (16,708 + 1/3) * 0.00000000001

which you can verify in your head, so the correct answer is clear.

Is it possible that the FoxPro calculation isn't what you describe?
If you use the tolerance 1.0045, a single digit off, then you get
0.00000167417, very close to your result.

-- Andy