+ Reply to Thread
Results 1 to 10 of 10

Efficiency Problem

  1. #1
    Registered User
    Join Date
    04-11-2011
    Location
    Dublin, Ireland
    MS-Off Ver
    Excel 2007
    Posts
    4

    Unhappy Efficiency Problem

    This is a long/tough one, so please bear with me. I will strive to be as clear as possible.

    I have created a workbook that does everything I want it to, but the problem is that it takes nearly 2 hours to re-calculate all of the formulae! I am hoping someone will be able to point out a few ways to make my formulae/layout more efficient.

    Details:
    Each month I export data fom a database into my workbook. It consists of 31 columns; each one containing a different ID or metric. Each row is an entry in the database. These entries are collected from 46 different sources each month, and each source can have multiple entries, as different metrics are collected at different points during the month. This leads to a sparce matrix of responses.

    One of the columns relates to the source. The sources are from a fixed list (ID1, ID2, ..., ID46) There is also a column relating to the corresponding month (May-11, June-11, ..., Dec-12)

    From this sparse matrix I wish to feed 46 spreadsheets, one for each source. I want to have the list of metrics in column A and the list of months across row 1 and then call on the export sheet to fill in the corresponding data.

    Method:
    Currently I use the following formula:

    {=INDEX(EXPORTDATA!$A:$AE,(MATCH($A$1&B$1,EXPORTDATA!$B:$B&EXPORTDATA!$AC:$AC,0)),XXX)}

    where: EXPORTDATA!$B:$B is the column relating to the source
    and: EXPORTDATA!$AC:$AC is the column relating to the month
    XXX relates to the column I wish to pull data from

    This formula works but it takes a very long time to calculate.

    I want it so that all I need to do each month is import the data and then all of the feilds automatically fill in. Since I wish to have a 2 year period, I get #N/A for most of my feilds, since there is no data present yet for the months in question. I simply use a conditional formatting to change the font of any errors to white so these errors are invisible.

    I then also want to have a TOTALS spreadsheet, fed by the 46 source sheets. Since there are so many #N/As in my workbook, simply summing the values does not work. I also cannot use a SUMIF formula due to the multiple sheets.

    Method:
    To get around this I use a very messy formula that utilises the IFERROR formula, wrapped up in an if statement to ensure any months that don't have data remain blank.

    =IF(SUM(IFERROR(ID1!E9,0),IFERROR(ID2!E9,0), ... ,IFERROR(ID46!E9,0))=0,"",SUM(IFERROR(ID1!E9,0),IFERROR(ID2!E9,0), ... ,IFERROR(ID46!E9,0)))

    This formula, again, works but slows the process down hugely when filled across and down all months and metrics.

    Any suggestion are more than welcome. Even for reading all the way through this, I thank you!

  2. #2
    Forum Guru MarvinP's Avatar
    Join Date
    07-23-2010
    Location
    Woodinville, WA
    MS-Off Ver
    Office 365
    Posts
    16,168

    Re: Efficiency Problem

    Hi cjki,

    This sounds like big and slow process. Trying to get data from other workbooks always takes a long time for them to be found, opened and merged. It seems you also have some Array Formulas (CSE formulas) which are volitile and take a long time to calculate.

    1. Have you tried to turn ScreenUpdating off?
    2. Have you tried to turn Calculation to off?
    Read http://www.cpearson.com/excel/optimize.htm

    There is a free add-on to Excel 2010 which claims to deal with multiple files called PowerPivots. I wonder if this would help solve your 2 hour process. Read about it at: http://www.powerpivot.com/

    Other than the above I think looking at the code and finding the slow spots would be the next step. Perhaps importing the data from each external file and then doing calculations on it from within the same workbook would improve performance. I think that doing a CSE using another workbooks data must slow things down a lot.

    hope this gives you some ideas.
    One test is worth a thousand opinions.
    Click the * Add Reputation below to say thanks.

  3. #3
    Registered User
    Join Date
    04-11-2011
    Location
    Dublin, Ireland
    MS-Off Ver
    Excel 2007
    Posts
    4

    Re: Efficiency Problem

    Hi Marvin,

    Thanks for your response. Unfortunately I am using Excel 2007 so I would not be able to use PowerPivots.

    I have screen updating and automatic calculation turned off when I am playing around with the formulae, but when it comes to recalculating the feilds after an import there is no avoiding the 2 hour wait!

    I was thinking the same as your point about trying to put everything on the same spreadsheet but there is so much data it would be very confusing for someone to find what they're looking for. I will try brainstorming new formulae to avoid the arrays I have. These arrays are used more than any othe formula, so replacing them would undoubtedly help.

    Thanks again!

  4. #4
    Forum Guru MarvinP's Avatar
    Join Date
    07-23-2010
    Location
    Woodinville, WA
    MS-Off Ver
    Office 365
    Posts
    16,168

    Re: Efficiency Problem

    Hi,
    Array formulas are volitile and take lots of time to calculate. I wonder if you can get the same answer using Pivot Tables instead of Array (CSE) formulas. That might help the speed.?

  5. #5
    Forum Expert shg's Avatar
    Join Date
    06-20-2007
    Location
    The Great State of Texas
    MS-Off Ver
    2003, 2010
    Posts
    40,678

    Re: Efficiency Problem

    Array formulas are volatile
    Array formulas are no more volatile than any other, Marvin. Volatile formulas arise from use of volatile functions, and the OP's formula has none.

    Currently I use the following formula:

    {=INDEX(EXPORTDATA!$A:$AE,(MATCH($A$1&B$1,EXPORTDATA!$B:$B&EXPORTDATA!$AC:$AC,0)),XXX)}
    If you are using that formula a lot, it's needlessly recomputing the red part every time, which requires a linear search. Put it in a separate cell, calculate it once, and reuse the result in a standard formula:

    =INDEX(EXPORTDATA!$A:$AE, thatCell, XXX)}
    Entia non sunt multiplicanda sine necessitate

  6. #6
    Forum Guru MarvinP's Avatar
    Join Date
    07-23-2010
    Location
    Woodinville, WA
    MS-Off Ver
    Office 365
    Posts
    16,168

    Re: Efficiency Problem

    Hey shg,

    I thought the CSE part of the formula with the "{" and "}" took lots of time to calculate. Isn't a CSE formula called Volitile too?

  7. #7
    Forum Expert shg's Avatar
    Join Date
    06-20-2007
    Location
    The Great State of Texas
    MS-Off Ver
    2003, 2010
    Posts
    40,678

    Re: Efficiency Problem

    I thought the CSE part of the formula with the "{" and "}" took lots of time to calculate.
    They can, and should be avoided where possible, but being volatile specifically means that they recalculate even when none of their precedents change.

    Charles Williams created this function to tell you if a formula is volatile:

    Please Login or Register  to view this content.
    E.g., =CalcCount(A1) and press F9. If the number increments, the cell has a volatile formula.

  8. #8
    Forum Expert martindwilson's Avatar
    Join Date
    06-23-2007
    Location
    London,England
    MS-Off Ver
    office 97 ,2007
    Posts
    19,320

    Re: Efficiency Problem

    maybe change to a non array version as well
    =INDEX(exportdata!$A1:$AE2000,MATCH($A$1&B$1,INDEX(exportdata!$B1:$B2000&exportdata!$AC1:$AC2000,0),0),5))
    "Unless otherwise stated all my comments are directed at OP"

    Mojito connoisseur and now happily retired
    where does code go ?
    look here
    how to insert code

    how to enter array formula

    why use -- in sumproduct
    recommended reading
    wiki Mojito

    how to say no convincingly

    most important thing you need
    Martin Wilson: SPV
    and RSMBC

  9. #9
    Forum Expert shg's Avatar
    Join Date
    06-20-2007
    Location
    The Great State of Texas
    MS-Off Ver
    2003, 2010
    Posts
    40,678

    Re: Efficiency Problem

    While that doesn't require CSE, Martin, I'm doubting that adding another function makes it calculate faster.

  10. #10
    Forum Expert martindwilson's Avatar
    Join Date
    06-23-2007
    Location
    London,England
    MS-Off Ver
    office 97 ,2007
    Posts
    19,320

    Re: Efficiency Problem

    hmm
    Calculation Overhead: Each time an array formula is calculated all of the virtual cells needed by the array formula are calculated, regardless of whether this is required or not. This may cause the array formula to be slower than a non-array equivalent set of formulae.
    but i'd have thought the same applies to the premise that match bit could be placed in another cell as you previously suggested.be interesting to try both and compare maybe?
    Last edited by martindwilson; 06-03-2011 at 07:53 PM.

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1