+ Reply to Thread
Results 1 to 4 of 4

What are the best practices for importing data with Power Query?

  1. #1
    Registered User
    Join Date
    01-05-2018
    Location
    Lexington, KY
    MS-Off Ver
    2007
    Posts
    28

    What are the best practices for importing data with Power Query?

    Apologies if this seem too broad or general, but I do have some specific questions.

    The place that I work at has a very poorly implemented specialty enterprise system, and the MRP (Material Resource Planning) module is pretty much useless. I with the help of this forum, I was able to design a bare bones "functioning" MRP system work around with Excel 2007 using existing reports from the ERP, and for that I am very grateful. However, I am not the type to to set it and forget it, and I my workbooks have grown unwieldy, sluggish, and well, they have never been pretty.

    So, while I can't really expect the company to spring for a $50k re-implementation of the ERP system, I was able to talk them into giving me one of the few computers that has an Excel 2013 key. Of course, this also comes at a time where I am tasked with generating more reports that the ERP system fails at. Frankly, I am just a couple of steps away from redesigning the entire ERP system output. With that in mind, I am turning my eye to the Power X suite of add-ins.

    So, per my question, what are some best practices? How do I accomplish this effectively?

    Some specific question I have:

    1. Importing Data:
      Ideally, I would like to query the SQL database directly, however our IT dept. consists of 1 person, and that person is actually our accountant. And while I have an AS in Comp Sci, I am a code monkey, not a network guru or DBA. So, my knowledge of SQL is limited, and I do not want to mess up our already shaky foundation. I have heard that even read-only access can cause issues if done incorrectly. Am I over thinking this? Please tell me that SQL queries aren't that scary, lol!

      The alternative is to build a "shadow" DB in Excel with exported reports. I don't mind maintaining such a DB, but I would like advise on building it if I have to go down that road. I already know that some data will need to maintain historical data, but generated reports will be snapshots that contain duplicate records. If I add these reports to the data source at a rate of ~600 records a week, 45 columns wide, how long before it becomes a performance problem? Will deleting out of date records cause problems later? What about daisy chaining reports into a "aggregate" table prior to utilization? Is it possible to import reports based on the folder they are in, and will it mess up if I remove files from that folder?

      Don't Do's? Mistake to avoid when preparing the data would definitely be helpful!
    2. Data Utilization
      So, this is more to do with the Power X suite. Namely, am I barking up the wrong tree by sticking to Excel, or should I make the jump to Power BI? I've read the pro's and con's, and feel like Excel is still the right choice but this is a lot of work, on top of all the work that I've done, and want to be certain that it is the right choice before committing. My understanding is that this is a matter of utility vs. aesthetics. I feel like more useful "work" will be done in the Excel environment rather than the BI environment, and the data will be coming from Excel already, so....

      I already know that once I get this ball rolling, the boss is going to want to see the data sliced every which way to Sunday... they already are asking more than I can give based on current utilization. however I like to be able to readily see my data sources, so that I know when something isn't kosher. Plus my impression is that Excel has more customization with regard to data manipulation. What is the compromise here?
    3. Resources
      Please, hook me up! Tutorials, blogs, videos... anything that you may feel is relevant or useful. Right now I am kind of aimlessly wandering in the world of data modelling.

    Well, I hope that this community can help me out once again. I'd like to apologize once again for the broad topic. Also, feel free to ask any questions about my projected use cases, I will be happy to answer!

    Thanks!
    dacheeba

  2. #2
    Forum Expert dflak's Avatar
    Join Date
    11-24-2015
    Location
    North Carolina
    MS-Off Ver
    365
    Posts
    7,920

    Re: What are the best practices for importing data with Power Query?

    I will say two words: MS-Query and Normalized Data.

    The attached word document explains the concept of normalized data This is the format in which you want to pull data down. Normalized data can be "attacked" by pivot tables efficiently.

    MS-Query is part of Excel 2013 and is IMHO the easiest way to make a data connection. The caveat is that you will need the appropriate driver for the database to which you are connecting. In my case, we connect to an Oracle database. You will have to establish this as an ODBC connection and MS-Query will recognize it.

    The first thing you should do when opening MS-Query is to turn off the wizard. It's more of a hindrance than a help. The next thing MS_Query offers is a list of tables in the database that you can select to build a query. The query builder is vaguely similar to the one in MS-Access. Again I recommend skipping this step and go directly to the design grid, select the SQL Icon and copy / paste your SQL code in there. Use a tool like WinSQL to debug the code. If it works in WinSQL it should work in MS-Query.

    I recommend keeping the SQL simple such as
    Please Login or Register  to view this content.
    You are right being concerned about read only access bogging down production systems. Keep the SQL simple and this is less likely to happen. Also it means that the query will be simple enough to be viewed graphically in MS-Query. Queries that can be presented graphically can get parameters such as StartDate and EndDate from Cells in Excel. So one of them could be =TODAY()-30 and the other simply = TODAY().

    The results will come back into Excel tables. See http://www.utteraccess.com/wiki/Tables_in_Excel for some of the benefits of Excel Tables if you are not already familiar with them.

    I can give you a hand with MS-Query if you need it. Step one is to get the driver (see your DBA about this) and set it up as an ODBC connection (I can help with this).

    Also once you get this technology in your corral, I have a system to run the data extraction and other reports using the Windows Task Scheduler which means you can run them in the middle of the night when the production floor is shut down, nobody is pounding on the database and traffic is light on the network. The best thing is, you come in in the morning and the reports for end of day yesterday are waiting for you.
    Attached Files Attached Files
    One spreadsheet to rule them all. One spreadsheet to find them. One spreadsheet to bring them all and at corporate, bind them.

    A picture is worth a thousand words, but a sample spreadsheet is more likely to be worked on.

  3. #3
    Forum Expert dflak's Avatar
    Join Date
    11-24-2015
    Location
    North Carolina
    MS-Off Ver
    365
    Posts
    7,920

    Re: What are the best practices for importing data with Power Query?

    Upon re-reading your requirements. If you can get CSV dumps for some of the tables, they can be concatenated and used as a source for MS-Query. MS-Query comes with a text driver. 600 records a week, 45 columns wide should not be an issue until your get years of data.

  4. #4
    Forum Moderator alansidman's Avatar
    Join Date
    02-02-2010
    Location
    Steamboat Springs, CO
    MS-Off Ver
    MS Office 365 Version 2405 Win 11 Home 64 Bit
    Posts
    23,882

    Re: What are the best practices for importing data with Power Query?

    You can also import tables and or queries directly from SQL server into Power Query. You can then manipulate the data in PQ and then load it into Excel for analysis. There are many tutorials on line on how to employ this powerful feature of Excel.
    Alan עַם יִשְׂרָאֵל חַי


    Change an Ugly Report with Power Query
    Database Normalization
    Complete Guide to Power Query
    Man's Mind Stretched to New Dimensions Never Returns to Its Original Form

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] Power Pivot does not load new column added in Power query
    By ibuhary in forum Excel General
    Replies: 12
    Last Post: 02-19-2019, 03:53 AM
  2. Sub-Forum for Excel Power Tools (Power Query, Power Pivot & Power BI)
    By chullan88 in forum Suggestions for Improvement
    Replies: 10
    Last Post: 06-28-2018, 02:25 PM
  3. Replies: 0
    Last Post: 04-05-2018, 01:16 AM
  4. Error problem, Importing table from pass protected workbook Power Query or other?
    By tta.akmal in forum Excel Programming / VBA / Macros
    Replies: 13
    Last Post: 03-29-2018, 03:59 AM
  5. SQL Query loading in MS Query but not importing data to excel
    By sharper1989 in forum Excel Programming / VBA / Macros
    Replies: 1
    Last Post: 05-20-2016, 09:52 AM
  6. [SOLVED] SQL Query loading in MS Query but not importing data to excel
    By sharper1989 in forum Access Tables & Databases
    Replies: 0
    Last Post: 05-19-2016, 09:31 AM
  7. Replies: 5
    Last Post: 05-09-2006, 01:25 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1