Re: rounding problems

From: Justin <justin(at)emproshunts(dot)com>
To: glene77is <glen(dot)e77is(at)gmail(dot)com>, pgsql-general(at)postgresql(dot)org
Subject: Re: rounding problems
Date: 2008-05-20 14:28:28
Message-ID: 4832E00C.3040506@emproshunts.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

glene77is wrote:
> On May 14, 3:27 pm, s(dot)(dot)(dot)(at)samason(dot)me(dot)uk (Sam Mason) wrote:
>
>> On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
>>
>>> Sam Mason wrote:
>>>
>>>> What doesfoxprouse for storing numbers? or is it just that you never
>>>> pushed it hard enough for the abstractions to show through.
>>>>
>>> I know i pushed it. Foxpro for the most has only 4 basic data types
>>> Numeric (similar to Posgresql numeric), Boolean, Date, Text aka
>>> (string) Thefoxprotables supported far more data types but when every
>>> it was dumped to variable it acted like one of the 4.
>>>
>> I really meant how much did you check the results, or did you accept
>> that they were correct?
>>
>>
>>> Foxprodid not suffer floating point math errors. I normally used 8 to
>>> 10 points precision. Foxprowas limited to 15 points of precision
>>> period. No more and no less, once you hit that was it.
>>>
>> 15 places seems very similar to what a 64bit IEEE floating point number
>> will give you, i.e. a double in C/C++.
>>
>>
>>> My problem is we calculate resistance of parts in aFoxproapp that we
>>> want to move because we want to bring all the custom apps into one
>>> framework and single database.
>>>
>>> Take this calculation (0.05/30000* 1.0025) which is used to calculate
>>> parts resistance and Tolerance. (its Ohms Law) The value returned from
>>> C++ = .0000016708 which is wrong
>>> it should be .00000167418. We just shrank the tolerance on the part we
>>> make
>>>
>> Why are you so sure about theFoxProresult? I've just checked a few
>> calculators and get results consistent with your C++ version.
>>
>> Justin C: 0.0000016708
>> JFoxPro: 0.00000167418
>> My C: 0.000001670833
>> bc[1]: 0.0000016708333333333333333333333333333332
>> PG[2]: 0.0000016708333333333333336675
>> Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
>>
>> Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
>> the math, and as they all agree I'm thinkingFoxProis incorrect! Next
>> I tried doing it accurately (in Haskell if it makes any difference) and
>> get an answer of 401/240000000 out, which would agree with everything
>> butFoxPro. If I calculate the ratio back out forFoxProI get
>> 401/239520242 which is a little way out.
>>
>>
>>> The Documentation from MS says 15 points of precision but the result say
>>> otherwise.
>>>
>> The docs for what?FoxProor their C compiler?
>>
>> If you meanFoxPro, I think this is another case of MS screwing up.
>>
>>
>>> I'm glad You and others are taking the time to explain to me
>>> the odd results before i get into redoing that application.
>>>
>> Welcome to the PG community, lots of people to get interested in lots of
>> things!
>>
>>
>>> Why oh Why did MS killFoxpro. :'( I understood it, knew its quirks
>>> and it worked very well with Postgresql
>>>
>> Are you sure you want to stay with it if its answers are wrong?
>>
>> Sam
>>
>
> *********************************************************************************
> This is fun, at 0400 AM. I enjoy reading Experts having serious fun!
>
> VFP 6.0, using my defaults
> ? (0.05/30000* 1.00250000000000000)
> displays "0l.000001670833333000"
>
> SET DECIMALS TO 15
> ? ((0.05/30000)* 1.0025)
> displays "0.000001670833333"
>
> and a frivolous example:
> SET DECIMALS TO 18
> ? ((0.050000/30000.00000000)* 1.0025000000000000)
> displays "0.000001670833333000"
>
Foxpro always stops at 15 decimals points, Even though some of the
documentation says 20 and 22 points of precision depending on the
version. I have versions 5 to 9
> Anybody tried to reckon this math
> the way we used to do it with a Slide-Rule ???
> (In VFP of course)
>
A slide what??. I have never touched one or seen a slide rule in real
life, just pretty pictures :-)
> glene77is
>

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Kevin Hunter 2008-05-20 14:30:22 [Fwd: Re: i am looking for postgresql hosting server]
Previous Message Karsten Hilbert 2008-05-20 14:14:19 Re: ranked subqueries vs distinct question