+ Reply to Thread
Results 1 to 2 of 2

Compare lists of URLs for duplicates, and remove duplicates

  1. #1
    Registered User
    Join Date
    06-13-2018
    Location
    Perth, Australia
    MS-Off Ver
    Office 365
    Posts
    1

    Compare lists of URLs for duplicates, and remove duplicates

    Hi everyone,

    First post here!

    For work at the moment I need to sort through a bunch of URLs, which may have duplicates in them in different formats, eg:

    They may start with:

    http
    https
    have www
    not have www
    have a trailing slash /
    have a trailing slash with a # at the end, eg. /#

    (Sorry not to be more specific, can't post links yet)

    These could sometimes be very long and URLs in general are all different lengths, but the bits I want to match should always ignore the final 2 characters in the url, and everything before the root domain on the site, eg. http, www, etc.

    I'd like to look for and delete duplicates, while still retaining one instance of the URL - which one doesn't matter so much, as they all the different variations point to the same content anyway.

    Is there any way, or any functions in excel I can use to make this as easy and hands-free as possible?

    Thanks!

  2. #2
    Forum Expert sandy666's Avatar
    Join Date
    02-05-2015
    Location
    Any Country
    MS-Off Ver
    farerwell
    Posts
    8,749

    Re: Compare lists of URLs for duplicates, and remove duplicates

    try to use PowerQuery aka Get&Transform (Data tab, Get&Transform section) which change urls to text, split url, then remove duplicates from appropriate column(s), after that merge columns with appropriate delimiteres and then compare result to source list (because you want working urls not a text - I suppose) - this is idea only but attached example excel file would help
    Last edited by sandy666; 06-13-2018 at 05:36 AM.
    sandy
    How to create an editor for Power Query with Notepad++ (tutorial)
    How to create timeline project with vertical today marker (2010, 2013, 2016 etc...) (examples)
    Tips for Excellent Spreadsheets

    What makes learning so hard is the amount of knowledge you have to unlearn
    Why is my program not doing what I expect?
    Because you set the wrong expectations. Rewire your brain

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Compare columns and remove duplicates
    By Goodstart14 in forum Excel Programming / VBA / Macros
    Replies: 2
    Last Post: 07-21-2015, 10:59 AM
  2. compare two short lists for duplicates
    By stockgoblin42 in forum Excel Formulas & Functions
    Replies: 4
    Last Post: 08-25-2013, 08:18 AM
  3. Remove duplicates from both Lists
    By Sircrayon in forum Excel General
    Replies: 1
    Last Post: 09-16-2010, 10:38 AM
  4. How to compare two columns and remove duplicates?
    By username123 in forum Excel General
    Replies: 15
    Last Post: 07-05-2006, 11:06 AM
  5. [SOLVED] how do i compare two columns and remove duplicates?
    By aljernon805 in forum Excel - New Users/Basics
    Replies: 1
    Last Post: 12-09-2005, 12:10 PM
  6. compare two columns and remove duplicates
    By Moni39 in forum Excel Formulas & Functions
    Replies: 3
    Last Post: 05-05-2005, 02:06 PM
  7. [SOLVED] How do I compare 2 lists and show duplicates as a new list?
    By AnaBannana in forum Excel Programming / VBA / Macros
    Replies: 3
    Last Post: 01-07-2005, 12:06 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1