+ Reply to Thread
Results 1 to 6 of 6

Problem with rounding

  1. #1
    Registered User
    Join Date
    07-15-2014
    Location
    Israel
    MS-Off Ver
    2010
    Posts
    3

    Problem with rounding

    Hi everyone,

    I have a set of functions that are executed one after another.
    The result is a value (A) and I need to subtract the value (B) from it. The final result is the value (A-B).

    The problem is that the value (A-B) that is the result of the subtraction is not exactly what I expect. It is larger by 0.0000000000001.

    I tried to isolate the values into a different sheet using "copy & paste as value" and trying to do the subtraction there, but the problem persists.

    Next I entered the values by hand (Manual Input) and it worked as expected.

    Can anyone tell my why this happens?

    Please see the attached sheet (just click where it says "Download: Weird.xlsx" at the top of the page)

  2. #2
    Forum Expert Olly's Avatar
    Join Date
    09-10-2013
    Location
    Darlington, UK
    MS-Off Ver
    Excel 2016, 2019, 365
    Posts
    6,284

    Re: Problem with rounding

    Well documented effect of floating point arithmetic.

    See http://support.microsoft.com/kb/214118
    let Source = #table({"Question","Thread", "User"},{{"Answered","Mark Solved", "Add Reputation"}}) in Source

    If I give you Power Query (Get & Transform Data) code, and you don't know what to do with it, then CLICK HERE

    Walking the tightrope between genius and eejit...

  3. #3
    Registered User
    Join Date
    07-15-2014
    Location
    Israel
    MS-Off Ver
    2010
    Posts
    3

    Re: Problem with rounding

    Thanks for the quick reply,

    The selected cell is the cell that I copy & pasted as value from the process. It appears to be a pure number (as seen in the formula tab) without any traces of functions. Is it still supposed to behave the way it does though?

    1zyf13s.jpg

  4. #4
    Forum Expert Olly's Avatar
    Join Date
    09-10-2013
    Location
    Darlington, UK
    MS-Off Ver
    Excel 2016, 2019, 365
    Posts
    6,284

    Re: Problem with rounding

    Yes. The link I posted above explains how to deal with it.

    This link explains in a bit more detail why it happens: http://support.microsoft.com/kb/78113

  5. #5
    Valued Forum Contributor
    Join Date
    07-07-2014
    Location
    Washington DC
    MS-Off Ver
    2007
    Posts
    1,047

    Re: Problem with rounding

    You can use the "=fixed" function to round your numbers down to one decimal place. Attached uses this function in two slightly different ways. Subsequent comparisons confirm the numbers match.
    Attached Files Attached Files

  6. #6
    Registered User
    Join Date
    07-15-2014
    Location
    Israel
    MS-Off Ver
    2010
    Posts
    3

    Re: Problem with rounding

    Thanks for the replies,

    I guess I'll have to make my code more robust using rounding and/or the "fixed" method.

    Cheers!

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Problem with Rounding
    By amartino44 in forum Excel Formulas & Functions
    Replies: 5
    Last Post: 05-23-2013, 04:58 PM
  2. Rounding Up & Down Problem
    By mrpunch in forum Excel General
    Replies: 4
    Last Post: 08-09-2011, 09:35 AM
  3. Rounding problem
    By p_dhoke in forum Excel General
    Replies: 2
    Last Post: 08-14-2007, 04:26 AM
  4. Rounding Problem
    By Brian O'Brien in forum Excel General
    Replies: 1
    Last Post: 06-12-2005, 12:05 AM
  5. rounding problem
    By bondcrash in forum Excel Programming / VBA / Macros
    Replies: 2
    Last Post: 04-25-2005, 02:06 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1