+ Reply to Thread
Results 1 to 4 of 4

How to easily find duplicate values in HUGE spreadsheet

  1. #1
    Registered User
    Join Date
    05-08-2015
    Location
    USA
    MS-Off Ver
    2010
    Posts
    21

    How to easily find duplicate values in HUGE spreadsheet

    I often run into this issue with things I do on a regular basis for work. I have an extremely large spreadsheet, for example 200,000 lines, and need to find the duplicate values. However, I don't just want to remove the duplicates, I need to see which values were duplicated so I can manually update them in a backend system. Normally I would use conditional formatting to highlight any duplicate values, then sort the column by cell color. When trying to do this on an enormous spreadsheet though, Excel completely freezes. Does anyone know of a better solution?

  2. #2
    Forum Moderator - RIP Richard Buttrey's Avatar
    Join Date
    01-14-2008
    Location
    Stockton Heath, Cheshire, UK
    MS-Off Ver
    Office 365, Excel for Windows 2010 & Excel for Mac
    Posts
    29,464

    Re: How to easily find duplicate values in HUGE spreadsheet

    Hi,

    Use a helper column to identify the duplicates. e.g.

    =COUNTIF(A$1:A$200000,A1)

    and copy down. Then filter on the helper column for values >1
    Richard Buttrey

    RIP - d. 06/10/2022

    If any of the responses have helped then please consider rating them by clicking the small star icon below the post.

  3. #3
    Administrator FDibbins's Avatar
    Join Date
    12-29-2011
    Location
    Duncansville, PA USA
    MS-Off Ver
    Excel 7/10/13/16/365 (PC ver 2310)
    Posts
    52,929

    Re: How to easily find duplicate values in HUGE spreadsheet

    Whatever you do with a file that big, may slow you down.

    Another option might be to use a helper column with COUNTIF(), then filter on all > 1?

    must be an echo in here...in here...here
    lol Richard
    1. Use code tags for VBA. [code] Your Code [/code] (or use the # button)
    2. If your question is resolved, mark it SOLVED using the thread tools
    3. Click on the star if you think someone helped you

    Regards
    Ford

  4. #4
    Registered User
    Join Date
    05-08-2015
    Location
    USA
    MS-Off Ver
    2010
    Posts
    21

    Re: How to easily find duplicate values in HUGE spreadsheet

    Thanks for the suggestion! I'm in the process of seeing if Excel can handle it. I wasn't familiar with COUNTIF so tested a small sheet first to make sure I got the results I wanted, and am currently waiting to see how long it takes to filter for the values greater than 1 in the actual document. I expect it will take forever, but hopefully filtering is at least somewhat quicker than sorting!

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] VBA macro that can find predetermined values within huge data set
    By mrsak87 in forum Excel Programming / VBA / Macros
    Replies: 5
    Last Post: 03-12-2015, 10:45 AM
  2. Adding new users easily to a spreadsheet
    By steelstorm in forum Excel Programming / VBA / Macros
    Replies: 2
    Last Post: 03-13-2012, 11:09 AM
  3. how to find autofilter columns in huge spreadsheet?
    By pleasehelpthanks in forum Excel General
    Replies: 1
    Last Post: 07-14-2011, 05:19 AM
  4. How to find duplicate in spreadsheet
    By rtanjci in forum Excel General
    Replies: 1
    Last Post: 07-07-2010, 01:00 AM
  5. Easily convert all formulas in 3d spreadsheet to values?
    By Jazza in forum Excel Formulas & Functions
    Replies: 2
    Last Post: 04-08-2006, 11:50 PM
  6. [SOLVED] Find duplicate numbers in large Excel Spreadsheet
    By Table in forum Excel Formulas & Functions
    Replies: 0
    Last Post: 09-06-2005, 07:05 AM
  7. Find duplicate numbers in large Excel Spreadsheet
    By Table in forum Excel Formulas & Functions
    Replies: 0
    Last Post: 09-06-2005, 04:05 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1