+ Reply to Thread
Results 1 to 5 of 5

decimal field of a number is changing when I strip off the integer field.

  1. #1
    Forum Contributor
    Join Date
    02-13-2016
    Location
    CT USA
    MS-Off Ver
    office 365 subscription
    Posts
    178

    decimal field of a number is changing when I strip off the integer field.

    lets assume I am trying to separate a number into its integer and decimal fields. Ie: 159.2345 into 159 and 0.2345

    When I separate the two fields, I am getting an incorrect number for the decimal field.

    Dim A as double
    Dim B as Integer
    Dim C as double

    A = 159.2345 In the debugger I can see that the decimal numbers do not extend beyond the 5
    B = int (A) THE Variable value of B is correct (159)
    C = A - B the Variable Value of C is incorrect (0.234499999999997)

    The result for C confuses me. the thousandths and 1/10 thousandths places are no longer 45
    I do not want to round C. this error ie revealing that I do not understand something.

    Perhaps it is because I am mixing data types. C = A(Double) - B(Integer)
    To test this theory out, I cast B into a Double but casting did not resolve my problem.

    Anyone have any ideas?

    I am running excel 365 home edition on W10 insider

    bil

  2. #2
    Forum Guru
    Join Date
    04-13-2005
    Location
    North America
    MS-Off Ver
    2002/XP and 2007
    Posts
    15,811

    Re: decimal field of a number is changing when I strip off the integer field.

    Looks like standard, routine double precision round-off error (you would probably get the exact same answer from any other programming language using double precision variables). https://en.wikipedia.org/wiki/Numeri...icrosoft_Excel


    Solution probably depends greatly on exactly what you are doing here.

    Considering that the error is much better than 1 part per trillion, exactly what is wrong with a result of 0.2344999999.....)? Is it a problem of testing for "exactly equal to"? or some other problem.

    In VBA, your calculation might be better performed using the currency data type (which uses up to 19 digits of precision, 15 to the left of the decimal point and 4 to the right) https://msdn.microsoft.com/en-us/lib.../gg264338.aspx If your number ever gets bigger than 15 digits, you will not be able to store that number in Excel as a number, though.

    Rounding functions might be appropriate, when needed.

    This is an inherent part of using numbers in a computer, so it could be important to be clear exactly what your problem is before deciding on the best approach to resolving that problem.
    Quote Originally Posted by shg
    Mathematics is the native language of the natural world. Just trying to become literate.

  3. #3
    Administrator 6StringJazzer's Avatar
    Join Date
    01-27-2010
    Location
    Tysons Corner, VA, USA
    MS-Off Ver
    MS365 Family 64-bit
    Posts
    24,701

    Re: decimal field of a number is changing when I strip off the integer field.

    The problem you are seeing is inherent in binary computers. Some decimal numbers cannot be represented exactly in binary, and it results in the kind of behavior you are seeing here.

    The usual way to deal with this is rounding to the desired precision.
    Jeff
    | | |會 |會 |會 |會 | |:| | |會 |會
    Read the rules
    Use code tags to [code]enclose your code![/code]

  4. #4
    Forum Expert shg's Avatar
    Join Date
    06-20-2007
    Location
    The Great State of Texas
    MS-Off Ver
    2003, 2010
    Posts
    40,678

    Re: decimal field of a number is changing when I strip off the integer field.

    When you subtract numbers, there is a loss of precision. The same thing occurs in the user interface:

    A
    B
    2
    159.234500000000000
    A2: Input
    3
    159.000000000000000
    A3: =INT(A2)
    4
    0.234499999999997
    A4: =A2-A3


    Round the fraction to the number of decimals required.
    Entia non sunt multiplicanda sine necessitate

  5. #5
    Forum Contributor
    Join Date
    02-13-2016
    Location
    CT USA
    MS-Off Ver
    office 365 subscription
    Posts
    178

    Re: decimal field of a number is changing when I strip off the integer field.

    Thank you for all your replies.

    Answers I read included:
    * problem you are seeing is inherent in binary computers
    * When you subtract numbers, there is a loss of precision
    * Looks like standard, routine double precision round-off error (you would probably get the exact same
    answer from any other programming language using double precision variables).

    I would like to present an argument that suggests your answers are not relevant to subtraction. They could be relevant to addition, division, multipliation, but not subtraction.

    Lets begin with a cursory look at the factors effecting arithmetic on computers.
    software stores floating point numbers in many ways. Storage is dependent on many factors that include hardware as well as chosen algorithm for handling floating point arithmetic. Those computers with hardware floating point processors encourage one kind of number storage and those computers without hardware floating point hardware can handle storage in multiple ways because there are many algorithms to chose from.

    One way, without a hardware accelerator is to separate the integer portion from the decimal.

    Lets take the number I presented to you as giving me my problem. 159.1234

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 6
    Last Post: 12-07-2014, 06:59 PM
  2. Changing the label of a Sum field in Values Field (DataField)
    By bezwlosy in forum Excel Charting & Pivots
    Replies: 5
    Last Post: 09-18-2014, 06:57 AM
  3. Strip a letter from the end of a numeric field
    By Olivepetunia in forum Excel Programming / VBA / Macros
    Replies: 5
    Last Post: 04-29-2013, 11:11 AM
  4. Changing a numerical field to a date field
    By MrSales in forum Excel Formulas & Functions
    Replies: 14
    Last Post: 12-29-2006, 02:33 PM
  5. [SOLVED] How can you strip the time of day out of a date field
    By Ron in forum Excel General
    Replies: 3
    Last Post: 04-19-2006, 02:30 PM
  6. Changing a text field to a date field
    By juliet in forum Excel - New Users/Basics
    Replies: 4
    Last Post: 02-21-2006, 05:55 PM
  7. Replies: 1
    Last Post: 10-24-2005, 06:05 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1