+ Reply to Thread
Results 1 to 4 of 4

Round function returns different values for same calculated input.

  1. #1
    Registered User
    Join Date
    06-16-2011
    Location
    NC, USA
    MS-Off Ver
    Excel 2003
    Posts
    4

    Round function returns different values for same calculated input.

    (A worksheet illustrating my issue is attached.)

    I've got 3 formulas that calculate to the same value (-0.05)

    = 58.1 - 58.15
    = 58.4 - 58.45
    = 64.4 - 64.45

    Why, then, does the round function return different values?

    =ROUND(58-58.05,1) returns 0
    =ROUND(58.4-58.45,1) returns -0.1
    =ROUND(64.4-64.45,1) returns 0

    Is there some logic at work here? (Maybe I'm searching the wrong keywords, but I'm stumped.) Thanks.
    Attached Files Attached Files
    Last edited by Kris Kohuth; 06-23-2011 at 11:06 PM.

  2. #2
    Forum Expert shg's Avatar
    Join Date
    06-20-2007
    Location
    The Great State of Texas
    MS-Off Ver
    2003, 2010
    Posts
    40,678

    Re: Round function returns different values for same calculated input.

    Binary floating point can't represent base 10 decimals precisely, so you get rounding errors, especially when you subtract.

    With the Double expressed in hex, -0.05 is BFA999999999999A; note that it's a repeating fraction (like 0.3333...) is in decimal.

    The results of your subtractions are

    Please Login or Register  to view this content.
    Entia non sunt multiplicanda sine necessitate

  3. #3
    Forum Guru MarvinP's Avatar
    Join Date
    07-23-2010
    Location
    Woodinville, WA
    MS-Off Ver
    Office 365
    Posts
    16,169

    Re: Round function returns different values for same calculated input.

    Hi Kris,
    Read http://www.cpearson.com/excel/rounding.htm then open you example workbook and
    click on F10 and find the Formula Evaluate Tool. Step through the evaluate tool and see how it deals with F10. Then do the same with F11 and F12.

    You will see that the internal representation of F10 and F12 are .4999999999972
    But in F11 the internal representation is .500000000043

    Then read the last section of http://office.microsoft.com/en-us/ex...010070518.aspx

    hope that also explains why .05 is not equal to .05 .
    One test is worth a thousand opinions.
    Click the * Add Reputation below to say thanks.

  4. #4
    Registered User
    Join Date
    06-16-2011
    Location
    NC, USA
    MS-Off Ver
    Excel 2003
    Posts
    4

    Re: Round function returns different values for same calculated input.

    Thanks to both of you. This forum is fantastic.

    You know, I saw that cpearson article and (sloppily) skipped over it, thinking it wouldn't apply to my values. My mistake.

    Thanks again.

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1