+ Reply to Thread
Results 1 to 5 of 5

Converting military time to decimal time not rounding correctly

  1. #1
    Forum Contributor
    Join Date
    03-03-2007
    Posts
    101

    Converting military time to decimal time not rounding correctly

    Attached is a spreadsheet that shows how Excel is not converting military time (hh:mm) to decimal hours correctly. Any clues as to why? One of the conversions is less than exact and it's throwing off my calculations.
    Attached Files Attached Files

  2. #2
    Registered User
    Join Date
    07-05-2006
    Posts
    59
    I got the calculations to work by changing the format of the two cells.

    See if that works for you . . . .
    Attached Files Attached Files

  3. #3
    Forum Guru
    Join Date
    08-05-2004
    Location
    NJ
    MS-Off Ver
    365
    Posts
    13,578
    What you are seeing, I believe, is the result of a machine that calculates in binary, trying to perform calculations on base 10 values. Sometimes it is not accurate out to the 16th decimal. You simply need to round your results using the ROUND function.

    ChemistB

  4. #4
    Forum Expert shg's Avatar
    Join Date
    06-20-2007
    Location
    The Great State of Texas
    MS-Off Ver
    2003, 2010
    Posts
    40,678
    The Chemist is spot on.

    In general, binary floating point numbers can't store decimals precisely. If you remember from high school, rational numbers (a/b) are always repeating fractions; for some, the repeating digit is 0, like 1/2 = 0.500000....

    For others, it's not zero: 1/9 = 0.1111111 ...

    For others, the repeating sequence is more than one digit: 3/7 = 0.428571 428571 428571 ...

    In binary, 0.1 (and lots of other decimal) have non-zero repeating binary decimals, and they are stored within the precision that the 64-bit IEEE-754 format allows.

    It's not an Excel thing.

  5. #5
    Forum Contributor
    Join Date
    03-03-2007
    Posts
    101
    That's interesting and complex. At least it's not Excel's fault. Thanks.

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1