+ Reply to Thread
Results 1 to 9 of 9

Counting Unique Values by Column - V LArge Dataset

  1. #1
    Registered User
    Join Date
    08-24-2016
    Location
    Berkshire, England
    MS-Off Ver
    2007
    Posts
    9

    Angry Counting Unique Values by Column - V LArge Dataset

    Hi,

    I am having some trouble using excel to count the number of unique values in a particular column of data.

    The data set I need to calculate this from is over 203,000 rows but filtered by year into subset of 86,000, 93,000 and the remaining.

    The column purely contains numbers and I need to know the number of unique values by year.

    I have tried filtering the dataset by another column which sets the year the data is from and then run the following formula =SUMPRODUCT((range<>"")/COUNTIF(range,range&"")) on the data for the relevant year.

    The problem is the amount of time the calculation is taking, I have started it on the 86,000 row data set and left it running for 3 days and it still hadn't completed the calculation.

    Can anyone a better more efficient way of working it out?

    Many Thanks
    Peter

  2. #2
    Forum Expert XOR LX's Avatar
    Join Date
    04-18-2013
    Location
    Turin, Italy
    MS-Off Ver
    Office 365
    Posts
    7,742

    Re: Counting Unique Values by Column - V LArge Dataset

    Hi,

    Because of this:

    Quote Originally Posted by PeterkMiles View Post
    The column purely contains numbers
    instead of this:

    =SUMPRODUCT((Range<>"")/COUNTIF(Range,Range&""))

    which is in any case a notoriously slow construction, you can use:

    =SUMPRODUCT(0+(FREQUENCY(Range,Range)>0))

    Even if we were talking about non-numerics, a FREQUENCY/MATCH set-up would still perform far, far better than the equivalent SUMPRODUCT/COUNTIF one, which, in my opinion, really should have been cast upon the scrapheap of redundant Excel constructions a long time back.

    Regards
    Click * below if this answer helped

    Advanced Excel Techniques: http://excelxor.com/

  3. #3
    Forum Expert Tony Valko's Avatar
    Join Date
    12-31-2011
    Location
    Pittsburgh
    MS-Off Ver
    2002, 2007:2013
    Posts
    18,890

    Re: Counting Unique Values by Column - V LArge Dataset

    Quote Originally Posted by XOR LX View Post
    =SUMPRODUCT((Range<>"")/COUNTIF(Range,Range&""))

    which is in any case a notoriously slow construction
    Agree, but that construct would be OK on "small" ranges.

    Just did a quick test on 65535 rows of numbers.

    =SUM(IF(FREQUENCY(A1:A65535,A1:A65535)>0,1)) (Normally entered, doesn't need CSE)

    =SUM(--(FREQUENCY(A1:A65535,A1:A65535)>0))

    =SUMPRODUCT(0+(FREQUENCY(A1:A65535,A1:A65535)>0))

    All 3 formulas averaged 6.5 secs to calculate.
    Biff
    Microsoft MVP Excel
    Keep It Simple Stupid

    Let's Go Pens. We Want The Cup.

  4. #4
    Forum Expert XOR LX's Avatar
    Join Date
    04-18-2013
    Location
    Turin, Italy
    MS-Off Ver
    Office 365
    Posts
    7,742

    Re: Counting Unique Values by Column - V LArge Dataset

    Thanks, Tony.

    Yes - perhaps with small ranges the SUMPRODUCT/COUNTIF set-up does not perform significantly worse than the equivalent FREQUENCY/MATCH one. But then why get into a bad habit?

    Also, you're correct that here SUMPRODUCT is not necessary - simply SUM will suffice.

    Regards

  5. #5
    Registered User
    Join Date
    08-24-2016
    Location
    Berkshire, England
    MS-Off Ver
    2007
    Posts
    9

    Re: Counting Unique Values by Column - V LArge Dataset

    Capture.PNG

    I must be doing something wrong, when using =SUMPRODUCT(0+(FREQUENCY(B2,B91376)>0)) it is only returning 1 unique entry but as can be seen from the screen shot there are 5 in view?

    What am I doing wrong?

  6. #6
    Registered User
    Join Date
    08-24-2016
    Location
    Berkshire, England
    MS-Off Ver
    2007
    Posts
    9

    Re: Counting Unique Values by Column - V LArge Dataset

    Capture.JPG

    sorry, try this attachement

  7. #7
    Forum Contributor
    Join Date
    06-30-2015
    Location
    Netherlands
    MS-Off Ver
    2013 / 2016 / 365
    Posts
    145

    Re: Counting Unique Values by Column - V LArge Dataset

    Hi,

    From my knowledge the SUM FREQUENCY option works fine but also has some calculation time.
    If you need the actual speed (let's say within 1 or 2 seconds) you need to switch to a VBA solution.
    HtH,

    Joris

    --------------------------------------------------------------------------------
    If you found my answer helpful, please hit the Like or Thank button.

    Please follow the forum Rules and Guidelines and use Code tags around your VBA code.

    Remember: test VBA code always on a copy of your file because usually you can't use undo

  8. #8
    Forum Expert Tony Valko's Avatar
    Join Date
    12-31-2011
    Location
    Pittsburgh
    MS-Off Ver
    2002, 2007:2013
    Posts
    18,890

    Re: Counting Unique Values by Column - V LArge Dataset

    Try it like this...

    =SUM(--(FREQUENCY(B2:B91376,B2:B91376)>0))

  9. #9
    Forum Expert XOR LX's Avatar
    Join Date
    04-18-2013
    Location
    Turin, Italy
    MS-Off Ver
    Office 365
    Posts
    7,742

    Re: Counting Unique Values by Column - V LArge Dataset

    Quote Originally Posted by PeterkMiles View Post
    I must be doing something wrong, when using =SUMPRODUCT(0+(FREQUENCY(B2,B91376)>0))
    You appear to be using just a single cell (B2) for the first argument.

    Have a look at my answer again:

    Quote Originally Posted by XOR LX View Post
    =SUMPRODUCT(0+(FREQUENCY(Range,Range)>0))
    Regards

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Deduping Values in Large Dataset - 7K rows X 30 columns
    By onespeed in forum Excel Formulas & Functions
    Replies: 7
    Last Post: 08-28-2015, 11:28 PM
  2. Replies: 3
    Last Post: 01-23-2014, 10:13 AM
  3. [SOLVED] counting unique values in one column for each value in another
    By seancain99 in forum Excel Formulas & Functions
    Replies: 5
    Last Post: 05-13-2013, 05:56 PM
  4. [SOLVED] counting unique values in col A against unique value in column B
    By greyscale in forum Excel Formulas & Functions
    Replies: 3
    Last Post: 05-08-2013, 08:43 AM
  5. [SOLVED] Counting unique-ish values in a column
    By D_N_L in forum Excel General
    Replies: 2
    Last Post: 07-25-2012, 04:44 AM
  6. Create New Worksheets based on one column in a Large Dataset
    By shaolinsnake in forum Excel Programming / VBA / Macros
    Replies: 1
    Last Post: 07-20-2012, 06:29 PM
  7. Vlookup for Multiple Values with Large Dataset
    By shanipk82 in forum Excel General
    Replies: 0
    Last Post: 02-06-2012, 05:11 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1