+ Reply to Thread
Results 1 to 5 of 5

Data Thinner code seems very inefficient- used on large data sets

  1. #1
    Registered User
    Join Date
    07-04-2013
    Location
    Calgary
    MS-Off Ver
    Excel 2010
    Posts
    36

    Data Thinner code seems very inefficient- used on large data sets

    Good afternoon!

    I have a file with two worksheets of data, one with 1000 lines and the other with 15,000 lines but both with 3 columns across. This is just a sample of the files I deal with, most run around 100,000 to 150,000 lines of data (three columns worth). I can manage these plots because my system has some power to it, however some other office systems do not have the same power and would like to run this plotting workbook I have created.

    I really want to thin this data from 150,000 to 15,000 (or something of that nature) and I figured I would write some code up to try and remove 9 of every 10 lines of data keeping that last one and moving to the next bundle of 9. This is what I came up with...

    Please Login or Register  to view this content.
    I thought I was being pretty clever but realized that even at 15,000 it is taking WAY too long and if people start using it for the 150,000 row file it would take over 10 mins (thats just on my computer).

    What am I doing wrong? Is there a way to thin the data better than what I am doing now?

    I have attached a stripped pile of data on sheet1 and sheet2 of the Data Thinner workbook with my macro inside if you are interested in looking at it.

    Thanks in advance!
    Attached Files Attached Files
    Last edited by Canuckle777; 07-25-2013 at 10:29 AM.

  2. #2
    Forum Guru Pete_UK's Avatar
    Join Date
    12-31-2011
    Location
    Warrington, England
    MS-Off Ver
    Office 2019 (still learning)
    Posts
    24,733

    Re: Data Thinner code seems very inefficient- used on large data sets

    Rather than delete small blocks continuously within a loop, it is better to just mark the rows for deletion within the loop and then select those marked rows and delete them all in one go (e.g. by using filters, or by sorting).

    Is your data sorted in some way, though? If not you could just delete the final 9/10ths of the records in one operation.

    Hope this helps.

    Pete

  3. #3
    Valued Forum Contributor
    Join Date
    03-21-2013
    Location
    cyberia
    MS-Off Ver
    Excel 2007
    Posts
    457

    Re: Data Thinner code seems very inefficient- used on large data sets

    try
    Please Login or Register  to view this content.

  4. #4
    Valued Forum Contributor
    Join Date
    03-21-2013
    Location
    cyberia
    MS-Off Ver
    Excel 2007
    Posts
    457

    Re: Data Thinner code seems very inefficient- used on large data sets

    or, if you don't need to retain formats, maybe a bit faster
    Please Login or Register  to view this content.

  5. #5
    Registered User
    Join Date
    07-04-2013
    Location
    Calgary
    MS-Off Ver
    Excel 2010
    Posts
    36

    Re: Data Thinner code seems very inefficient- used on large data sets

    Kalak, this is masterful. It totally worked and it handles 150,000 lines of data like a champ. No more waiting 15 mins. You rock!

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. data problems, matching large data sets with smaller known points
    By awguest1 in forum Excel Formulas & Functions
    Replies: 4
    Last Post: 05-28-2013, 08:48 PM
  2. [SOLVED] How to merge two data sets to create a single large data set.
    By Econocrat in forum Excel General
    Replies: 5
    Last Post: 10-06-2012, 04:02 PM
  3. interrogating large data sets
    By ruleworld in forum Excel General
    Replies: 1
    Last Post: 01-04-2011, 07:13 AM
  4. Inefficient/slow data cleanup code
    By teitoku in forum Excel Programming / VBA / Macros
    Replies: 16
    Last Post: 12-01-2010, 06:59 PM
  5. Data matching from 2 large data sets
    By chemnerd1 in forum Excel Programming / VBA / Macros
    Replies: 4
    Last Post: 10-31-2008, 04:22 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1