+ Reply to Thread
Results 1 to 11 of 11

Isolating Duplicates - Better Method Than Conditional Format and Filter?

  1. #1
    Registered User
    Join Date
    09-26-2012
    Location
    Pittsburgh, PA
    MS-Off Ver
    MS Office 2010
    Posts
    64

    Isolating Duplicates - Better Method Than Conditional Format and Filter?

    I'm working on isolating duplicate values in a huge list of products (Over 200k rows). I'm familiar with conditional formatting to highlight duplicates, and when I try to filter on those, it bogs down my machine given the massive size of the file. Is there a faster/better way to find/isolate duplicate values? Perhaps something that would even pull them into a new spreadsheet?

    Any advice/pointers would be greatly appreciated.

    Thanks

  2. #2
    Forum Guru TMS's Avatar
    Join Date
    07-15-2010
    Location
    The Great City of Manchester, NW England ;-)
    MS-Off Ver
    MSO 2007,2010,365
    Posts
    44,918

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    What constitutes a duplicate?


    Regards, TMS
    Trevor Shuttleworth - Retired Excel/VBA Consultant

    I dream of a better world where chickens can cross the road without having their motives questioned

    'Being unapologetic means never having to say you're sorry' John Cooper Clarke


  3. #3
    Registered User
    Join Date
    09-26-2012
    Location
    Pittsburgh, PA
    MS-Off Ver
    MS Office 2010
    Posts
    64

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    Same spelling. Numeric or alphanumeric. So, for example, it could be ABC123EDF. If that shows up twice in the same column, I'd like to isolate it.

  4. #4
    Forum Guru Jonmo1's Avatar
    Join Date
    03-08-2013
    Location
    Bryan, TX
    MS-Off Ver
    Excel 2010
    Posts
    9,763

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    If your goal is to actually 'remove' the duplicates, then there is a very nice "Remove Duplicates" feature on the Data tab.

    But actually identifying/flagging the duplicates is more difficult as you can see.

    Another method might be to use a standard cell formula instead of conditional formatting..

    In an adjescent column filled to the end.
    =IF(COUNTIF(A$1:A1,A1)>1,"it's a dupe","")

    Then filter on that column.


    I don't know if that's any better than the current conditional formatting method..

  5. #5
    Forum Contributor
    Join Date
    08-08-2005
    Location
    Kansas, USA
    MS-Off Ver
    2016
    Posts
    293

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?


  6. #6
    Registered User
    Join Date
    09-26-2012
    Location
    Pittsburgh, PA
    MS-Off Ver
    MS Office 2010
    Posts
    64

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    I'll try the IF/COUNTIF formula and see if that speeds up the process. Not trying to remove the duplicates, definitely need to flag them.

  7. #7
    Forum Guru TMS's Avatar
    Join Date
    07-15-2010
    Location
    The Great City of Manchester, NW England ;-)
    MS-Off Ver
    MSO 2007,2010,365
    Posts
    44,918

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    This will identify duplicates in column A:

    Cell A2: =COUNTIFS($A$2:$A2,$A2)

    and copy down. Anything not equal to 1 is a duplicated record. Anything with a 1 is either unique or the first record for that key. Logically, you could, therefore, remove all but one of the entries.


    Regards, TMS


    Edit: corrected formula
    Last edited by TMS; 07-31-2013 at 12:04 PM. Reason: edit formula

  8. #8
    Registered User
    Join Date
    09-26-2012
    Location
    Pittsburgh, PA
    MS-Off Ver
    MS Office 2010
    Posts
    64

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    Is it possible to modify one of those formulas so that it flags both occurrences of the duplicate as being a duplicate, instead of just the second occurrence? I want to be able to select both so that I can determine which to remove based on data in other columns.

  9. #9
    Forum Guru Jonmo1's Avatar
    Join Date
    03-08-2013
    Location
    Bryan, TX
    MS-Off Ver
    Excel 2010
    Posts
    9,763

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    Just make the range encompass the whole set of data.
    But this will decrease the performance

    =IF(COUNTIF(A$1:A$200000,A1)>1,"it's a dupe","")

  10. #10
    Forum Guru TMS's Avatar
    Join Date
    07-15-2010
    Location
    The Great City of Manchester, NW England ;-)
    MS-Off Ver
    MSO 2007,2010,365
    Posts
    44,918

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    It's not a competition, but this formula:

    =COUNTIFS($A$2:$A2,$A2)

    numbers the records in the order they appear in the file. That might indicate some sort of priority.

    Regards, TMS
    Last edited by TMS; 07-31-2013 at 12:05 PM. Reason: edit formula

  11. #11
    Registered User
    Join Date
    03-14-2011
    Location
    California, USA
    MS-Off Ver
    Excel 2003
    Posts
    6

    Re: Isolating Duplicates - Better Method Than Conditional Format and Filter?

    If your goal is to actually 'remove' the duplicates, then there is a very nice "Remove Duplicates" feature on the Data tab.

    This was exactly what I needed. Thanks!

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Excel 2007 : Conditional Format + Filter
    By pansovic in forum Excel General
    Replies: 2
    Last Post: 08-19-2011, 11:21 AM
  2. Isolating Duplicates Among 2 Large Pieces of Data
    By Wonton in forum Excel General
    Replies: 5
    Last Post: 04-02-2011, 10:11 AM
  3. Conditional Format Duplicates
    By jodyatspd in forum Excel Formulas & Functions
    Replies: 1
    Last Post: 12-05-2008, 09:51 AM
  4. Conditional Format Duplicates
    By tedwood in forum Excel General
    Replies: 3
    Last Post: 10-15-2007, 05:19 AM
  5. [SOLVED] conditional format for duplicates
    By PamHR in forum Excel Formulas & Functions
    Replies: 2
    Last Post: 10-22-2005, 08:05 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1