+ Reply to Thread
Results 1 to 2 of 2

Strange Decimal Values? Rounding Error?

  1. #1
    Registered User
    Join Date
    06-01-2013
    Location
    United States
    MS-Off Ver
    Excel 2011
    Posts
    64

    Strange Decimal Values? Rounding Error?

    So I'm having this strange problem. In the picture you should see a small table 3x4 in size. Each value in column 1 is multiplied by 2. Column 2 is multiplied by 5, column 3 is multiplied by 10 and column 4 is multiplied by 0.

    So when I multiply .4 by 2...I expect .8 and not .800001
    Now, this doesn't make much of a difference but I'd prefer it to not do that.

    Likewise when I multiply .9 by 10, I'd like to get 9. While this is very close to 9, I'm not sure why this is happening.
    I looked through preferences and found no obvious answer.

    What's going on?
    Thanks.


    Screen Shot 2013-06-20 at 12.51.12 PM.pngScreen Shot 2013-06-20 at 12.52.27 PM.png

  2. #2
    Forum Expert shg's Avatar
    Join Date
    06-20-2007
    Location
    The Great State of Texas
    MS-Off Ver
    2003, 2010
    Posts
    40,678

    Re: Strange Decimal Values? Rounding Error?

    The binary floating point format your computer uses can't store most base 10 decimals exactly, so errors accumulate. Use ROUND().
    Entia non sunt multiplicanda sine necessitate

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1