Maths using Double type

t is the Floating Point processor.

Try this to see what is happening

Please Login or Register  to view this content.

Hi Bob,

Thanks for the extra line of code which now confuses me even more...
Somewhere along the lines you would think maths would apply?
My code is basically saying if (3 * 0.2) = 0.6 then... Which it does.
Now you have me doing something even more bizaar.
(3 * 0.2) - 0.6 = 0.. right? No according to result I am getting its 5.55111512312578E-17

Anyone able to explain what floating point precision is and why it affects simple maths.
Where I am sitting just now is I cannot trust Excel to do simple maths. 3 * 0.2. Simples. No?

Have a read of this http://support.microsoft.com/kb/42980

As I mentioned, this is one of the side effects of floating point arithmetic, using a binary engine for decimal calculations. This post might help with an understanding http://support.microsoft.com/kb/42980

(admin, he has added code tags, so he is a good boy now).

Originally Posted by romperstomper

@snb,
How does converting to a long integer help in any way? Using your code, if you change FirstNumber to 0.7 that also "works". As does 1.2 or anything else that rounds to 1.

It's called obfuscation!

[QUOTE=Bob Phillips;2607705]As I mentioned, this is one of the side effects of floating point arithmetic, using a binary engine for decimal calculations. This post might help with an understanding http://support.microsoft.com/kb/42980

Hi Bob, thanks for the link which does indeed explain it very well. I shall be more careful in future when using the Double type. I always use it when really a Single will do in most cases. Old habit needs broken. Thanks for the guidance.

Mental note... when comparing results if Double is required then I shall CStr(result) to compare values.... This works in this example but I could not understand why.. Now I do.

Hi Colin,

I have no idea why CStr works but it does...

If you try my orginal post you will see the msgbox on not matching provides the following output Not Worked >0.6<>0.6<.
This means the value of SecondNumber * 0.2 = 0.6 and firstNumber also = 0.6.

So when I cast both sides as string this matches and I do not have the fault.....

Hi Colin,

Yes I can see why comparing numerics as strings could be a threat but not when using equals. In the application I am using things are either an exact match or not so I am comfortable in using CStr(myval) = Cstr(testResult).