+ Reply to Thread
Results 1 to 11 of 11

Large data set and complexish formulae causing system to be slow

  1. #1
    Registered User
    Join Date
    04-24-2024
    Location
    London
    MS-Off Ver
    365
    Posts
    7

    Large data set and complexish formulae causing system to be slow

    Hello,

    I have what seems to be a larger than usual file 42MB, with the main sheet containing 34000 rows and 162 columns. The data is plain data with no conditional formatting. I add about 500 rows every week.?*

    To analyse the data, I use a separate sheet in the same workbook, with a bunch of formulae. All theses formulae, 40 in total, are conditional formulae referencing whole columns. Here is an example:?*

    =LET(
    A,FTSAdvanced!$AE:$AE,
    B,FTSAdvanced!$AI:$AI,
    C,FTSAdvanced!$BB:$BB,
    D,FTSAdvanced!$BF:$BF,
    E,FTSAdvanced!$BX:$BX,
    H,FTSAdvanced!$EH:$EH,
    I,FTSAdvanced!$EQ:$EQ,
    J,FTSAdvanced!$CX:$CX,
    K,FTSAdvanced!$CO:$CO,
    L,FTSAdvanced!$DG:$DG,
    M,FTSAdvanced!$D:$D,
    MAX(FREQUENCY(IF(
    (IF(OR($B$2="",$C$2=""),1,(A>=$B$2)*(A<=$C$2)))*
    (IF(OR($B$3="",$C$3=""),1,(B>=$B$3)*(B<=$C$3)))*
    (IF(OR($B$4="",$C$4=""),1,(C>=$B$4)*(C<=$C$4)))*
    (IF(OR($B$5="",$C$5=""),1,(D>=$B$5)*(D<=$C$5)))*
    (IF(OR($B$6="",$C$6=""),1,(E>=$B$6)*(E<=$C$6)))*
    (IF(OR($B$9="",$C$9=""),1,(H>=$B$9)*(H<=$C$9)))*
    (IF(OR($B$10="",$C$10=""),1,(I>=$B$10)*(I<=$C$10)))*
    (IF(OR($B$11="",$C$11=""),1,(J>=$B$11)*(J<=$C$11)))*
    (IF(OR($B$12="",$C$12=""),1,(K>=$B$12)*(K<=$C$12)))*
    (IF(OR($B$13="",$C$13=""),1,(L>=$B$13)*(L<=$C$13)))*(M=$A$1)
    *(FTSAdvanced!$FL:$FL="N"),ROW(FTSAdvanced!$FL:$FL)),
    IF(
    (IF(OR($B$2="",$C$2=""),1,(A>=$B$2)*(A<=$C$2)))*
    (IF(OR($B$3="",$C$3=""),1,(B>=$B$3)*(B<=$C$3)))*
    (IF(OR($B$4="",$C$4=""),1,(C>=$B$4)*(C<=$C$4)))*
    (IF(OR($B$5="",$C$5=""),1,(D>=$B$5)*(D<=$C$5)))*
    (IF(OR($B$6="",$C$6=""),1,(E>=$B$6)*(E<=$C$6)))*
    (IF(OR($B$9="",$C$9=""),1,(H>=$B$9)*(H<=$C$9)))*
    (IF(OR($B$10="",$C$10=""),1,(I>=$B$10)*(I<=$C$10)))*
    (IF(OR($B$11="",$C$11=""),1,(J>=$B$11)*(J<=$C$11)))*
    (IF(OR($B$12="",$C$12=""),1,(K>=$B$12)*(K<=$C$12)))*
    (IF(OR($B$13="",$C$13=""),1,(L>=$B$13)*(L<=$C$13)))*(M=$A$1)
    *(FTSAdvanced!$FL:$FL<>"N"),
    ROW(FTSAdvanced!$FL:$FL)
    ))))

    My concern is that I need to add more formulae to be able to analyse the data properly but excel is taking more and more time to calculate - 30 to 40 seconds. I am not an excel pro, as you would have guessed, and I would really appreciate some help to make excel work more efficiently.?*

    As I said, there isn't any conditional formatting, VBA or volatile formulae. Calculating is in manual mode, and everything is saved on my machine.

    My questions are:

    How could I make those formulae more efficient?

    Is there an alternative to referencing the whole columns? Should I use dynamic ranges - index? If so will this help with speed?

    Should I convert my data into a huge table? Would this make excel work easier?

    I run excel 365, on an inter mac 16gb. I can't access power pivot or anything other than what I already have.

    Thanks

    Excel help 2504.xlsx
    Last edited by NickSe34; 04-25-2024 at 09:45 AM.

  2. #2
    Forum Moderator Glenn Kennedy's Avatar
    Join Date
    07-08-2012
    Location
    Digital Nomad... occasionally based in Ireland.
    MS-Off Ver
    O365 (PC) V 2403
    Posts
    44,116

    Re: Large data set and complexish formulae causing system to be slow

    A guess... referencing whole columns will be the most likely cause. If you have 40000 rows, Excel may waste a lot of time calculating 1,000,000++ rows over, and over again
    Glenn




    None of us get paid for helping you... we do this for fun. So DON'T FORGET to say "Thank You" to all who have freely given some of their time to help YOU

  3. #3
    Registered User
    Join Date
    04-24-2024
    Location
    London
    MS-Off Ver
    365
    Posts
    7
    Quote Originally Posted by Glenn Kennedy View Post
    A guess... referencing whole columns will be the most likely cause. If you have 40000 rows, Excel may waste a lot of time calculating 1,000,000++ rows over, and over again
    What do you suggest? Dynamic range names? Using index in my formulae? Helper cells? Any suggestions?

  4. #4
    Forum Moderator Glenn Kennedy's Avatar
    Join Date
    07-08-2012
    Location
    Digital Nomad... occasionally based in Ireland.
    MS-Off Ver
    O365 (PC) V 2403
    Posts
    44,116

    Re: Large data set and complexish formulae causing system to be slow

    Probably convert it to a table and use structured Table references. However the whole thing looks unwieldy!! My tiny brain can't possibly figure out what it's doing... but I'd bet you a shiny penny that there's a better way to do it!!

    Any chance of getting a SMALL sample sheet?

  5. #5
    Forum Moderator Glenn Kennedy's Avatar
    Join Date
    07-08-2012
    Location
    Digital Nomad... occasionally based in Ireland.
    MS-Off Ver
    O365 (PC) V 2403
    Posts
    44,116

    Re: Large data set and complexish formulae causing system to be slow

    Small = 10-20 rows... not thousands... to illustrate the PRINCIPLE...

  6. #6
    Forum Guru TMS's Avatar
    Join Date
    07-15-2010
    Location
    The Great City of Manchester, NW England ;-)
    MS-Off Ver
    MSO 2007,2010,365
    Posts
    44,521

    Re: Large data set and complexish formulae causing system to be slow

    An example of limiting the row count without converting the data to a Table.

    Formula: copy to clipboard
    Please Login or Register  to view this content.


    Ideally, you should use the same (fully populated) column to determine the last row ( lr ) for all ranges. I've just used AE in the absence of any data.
    Trevor Shuttleworth - Retired Excel/VBA Consultant

    I dream of a better world where chickens can cross the road without having their motives questioned

    'Being unapologetic means never having to say you're sorry' John Cooper Clarke


  7. #7
    Registered User
    Join Date
    04-24-2024
    Location
    London
    MS-Off Ver
    365
    Posts
    7
    Quote Originally Posted by Glenn Kennedy View Post
    Probably convert it to a table and use structured Table references. However the whole thing looks unwieldy!! My tiny brain can't possibly figure out what it's doing... but I'd bet you a shiny penny that there's a better way to do it!!

    Any chance of getting a SMALL sample sheet?
    I am pretty sure there is a better way to do it. I’ll share a sample in the morning. Thanks

  8. #8
    Registered User
    Join Date
    04-24-2024
    Location
    London
    MS-Off Ver
    365
    Posts
    7
    Quote Originally Posted by TMS View Post
    An example of limiting the row count without converting the data to a Table.

    Formula: copy to clipboard
    Please Login or Register  to view this content.


    Ideally, you should use the same (fully populated) column to determine the last row ( lr ) for all ranges. I've just used AE in the absence of any data.
    Thanks for your suggestions. I didn’t know about the use of lr. I’ll share a sample in the morning and explain a bit more.

  9. #9
    Registered User
    Join Date
    04-24-2024
    Location
    London
    MS-Off Ver
    365
    Posts
    7

    Re: Large data set and complexish formulae causing system to be slow

    Hi Glenn, I have shared a sample sheet with some data on 1 sheet and the conditions and some formula on Sheet 2. If you can have a look that's be great. Thanks for your time.

  10. #10
    Registered User
    Join Date
    04-24-2024
    Location
    London
    MS-Off Ver
    365
    Posts
    7

    Re: Large data set and complexish formulae causing system to be slow

    Quote Originally Posted by Glenn Kennedy View Post
    Probably convert it to a table and use structured Table references. However the whole thing looks unwieldy!! My tiny brain can't possibly figure out what it's doing... but I'd bet you a shiny penny that there's a better way to do it!!

    Any chance of getting a SMALL sample sheet?
    Hi Glenn, I have uploaded the file. Thanks for helping. Nick

  11. #11
    Registered User
    Join Date
    04-24-2024
    Location
    London
    MS-Off Ver
    365
    Posts
    7

    Re: Large data set and complexish formulae causing system to be slow

    Quote Originally Posted by TMS View Post
    An example of limiting the row count without converting the data to a Table.

    Formula: copy to clipboard
    Please Login or Register  to view this content.


    Ideally, you should use the same (fully populated) column to determine the last row ( lr ) for all ranges. I've just used AE in the absence of any data.
    Hi TMS, I have shared a sample of my data and some of the formula. Could you help me understand theformula you've shared? The formula in my orginal post looks for the longest consecutive run of "N" in the column FL:FL, depending on the conditions:
    A,FTSAdvanced!$AE:$AE,
    B,FTSAdvanced!$AI:$AI,
    C,FTSAdvanced!$BB:$BB,
    D,FTSAdvanced!$BF:$BF,
    E,FTSAdvanced!$BX:$BX,
    H,FTSAdvanced!$EH:$EH,
    I,FTSAdvanced!$EQ:$EQ,
    J,FTSAdvanced!$CX:$CX,
    K,FTSAdvanced!$CO:$CO,
    L,FTSAdvanced!$DG:$DG,
    M,FTSAdvanced!$D:$D.

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] VBA script is using ALL rows in worksheet causing large file size and slow performance
    By eyanosa in forum Excel Programming / VBA / Macros
    Replies: 4
    Last Post: 01-10-2023, 02:21 PM
  2. Very slow process, sorting large table to identify range to apply array formulae
    By jasonmcasey in forum Excel Programming / VBA / Macros
    Replies: 2
    Last Post: 05-15-2015, 03:40 AM
  3. Replies: 10
    Last Post: 10-25-2012, 03:31 PM
  4. System is really really slow during data entry...VBA problem?
    By Lifeseeker in forum Excel Programming / VBA / Macros
    Replies: 3
    Last Post: 12-30-2011, 06:37 PM
  5. Lots of data causing slow calculations
    By penfold in forum Excel Formulas & Functions
    Replies: 4
    Last Post: 05-22-2008, 05:22 PM
  6. [SOLVED] AutoFilter Best Practice when used in large files (slow system dow
    By Dennis in forum Excel General
    Replies: 2
    Last Post: 02-17-2006, 03:55 PM
  7. [SOLVED] excel causing system to be in low system resource
    By inenewbl in forum Excel General
    Replies: 0
    Last Post: 04-05-2005, 12:06 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1