+ Reply to Thread
Results 1 to 11 of 11

Conditional remove duplicates based on another column in power query related to (power bi)

  1. #1
    Registered User
    Join Date
    08-29-2018
    Location
    Melbourne, Australia
    MS-Off Ver
    2016/365 both
    Posts
    22

    Conditional remove duplicates based on another column in power query related to (power bi)

    Hello
    I'm new to Power Bi, and realise my power query tables do not have distinct rows. I would like to delete the duplicate rows based on the lower FTE after first summing the FTE. I have about 20 staff who have duplicate rows out of 150 staff, but I can't get the source data (SQL database) to change to alleviate this problem.

    My attached small table shows my duplicate values and how I would like it to look so I can use if for a Power BI dashboard. Even if I could be guided towards some PD I would appreciate that. So far my google search has been fruitless.
    Attached Files Attached Files

  2. #2
    Forum Moderator alansidman's Avatar
    Join Date
    02-02-2010
    Location
    Steamboat Springs, CO
    MS-Off Ver
    MS Office 365 Version 2404 Win 11 Home 64 Bit
    Posts
    23,865

    Re: Conditional remove duplicates based on another column in power query related to (power

    In Power Query, here is the Mcode

    Please Login or Register  to view this content.
    Basically, highlight first column (ID)
    With Home Tab open, click on Remove Rows
    Select Remove Duplicates.
    Alan עַם יִשְׂרָאֵל חַי


    Change an Ugly Report with Power Query
    Database Normalization
    Complete Guide to Power Query
    Man's Mind Stretched to New Dimensions Never Returns to Its Original Form

  3. #3
    Registered User
    Join Date
    08-29-2018
    Location
    Melbourne, Australia
    MS-Off Ver
    2016/365 both
    Posts
    22

    Re: Conditional remove duplicates based on another column in power query related to (power

    Hi Alan
    Thanks for replying.
    I don't think I was clear. I would like the duplicated row to be deleted after it has summed the FTE value of both rows, pertaining to the same ID (Staff member) and then the row remaining is the higher fte row. The order of each row for the same employee may be random, so deleting the first or second row won't work either. Hopefully that helps with my predicament.

  4. #4
    Forum Moderator alansidman's Avatar
    Join Date
    02-02-2010
    Location
    Steamboat Springs, CO
    MS-Off Ver
    MS Office 365 Version 2404 Win 11 Home 64 Bit
    Posts
    23,865

    Re: Conditional remove duplicates based on another column in power query related to (power

    Got it. Look at the file now. I grouped the IDs so that the FTE is summed and then removed duplicates. See if this works for you.
    Attached Files Attached Files

  5. #5
    Registered User
    Join Date
    08-29-2018
    Location
    Melbourne, Australia
    MS-Off Ver
    2016/365 both
    Posts
    22

    Re: Conditional remove duplicates based on another column in power query related to (power

    Thanks Alan
    It works for merging the IDs and totalling the FTE. However, it isn't sensitive to the higher FTE being the distinct row I need. i.e. if I reverse the fte for ID 6111 it still gives me the Teacher instead of the ES officer. Unfortunately, I need the higher FTE due to Leave records being included at 0 FTE which I don't want to remain.
    Does that make sense? I may need some sort of max formula in there!!!

  6. #6
    Forum Moderator alansidman's Avatar
    Join Date
    02-02-2010
    Location
    Steamboat Springs, CO
    MS-Off Ver
    MS Office 365 Version 2404 Win 11 Home 64 Bit
    Posts
    23,865

    Re: Conditional remove duplicates based on another column in power query related to (power

    Sorry, I am out of Options. Will put in call for others to see if they have any ideas. For the future, it is important and very helpful if you express all criteria at the onset instead of piecemeal. After all, what is clear in your mind and not expressed cannot be mind read by us.

  7. #7
    Registered User
    Join Date
    08-29-2018
    Location
    Melbourne, Australia
    MS-Off Ver
    2016/365 both
    Posts
    22

    Re: Conditional remove duplicates based on another column in power query related to (power

    Thanks for trying Alan.

  8. #8
    Forum Moderator AliGW's Avatar
    Join Date
    08-10-2013
    Location
    Retired in Ipswich, Suffolk, but grew up in Sawley, Derbyshire (England)
    MS-Off Ver
    MS 365 Subscription Insider Beta Channel v. 2404 (Windows 11 22H2 64-bit)
    Posts
    80,460

    Re: Conditional remove duplicates based on another column in power query related to (power

    Here's my take on it.

    M Code:

    Please Login or Register  to view this content.
    Excel 2016 (Windows) 32 bit
    A
    B
    C
    D
    E
    F
    G
    H
    I
    J
    K
    L
    17
    ID JobPositionCode AwardCode AwardLevelCode JobPositionCategoryCode JobPositionActiveFlag Surname Preferred FTE CurrentJobPositionFlag FutureJobPositionFlag JobStartDate
    18
    5000
    PRINCIPAL TEACHING LEADERSHIP TEACHER
    TRUE
    Wilson Rebel
    1.00
    TRUE
    FALSE
    28/11/2018 00:00
    19
    6111
    TEACHER TEACHING COACH TEACHER
    TRUE
    Blanchett Cate
    0.80
    TRUE
    FALSE
    31/03/2016 00:00
    20
    25000
    TEACHER TEACHING HEAD OF HOUSE TEACHER
    TRUE
    Jackman Hugh
    1.00
    TRUE
    FALSE
    13/01/2020 00:00
    21
    6000
    ES FINANCE PAYROLL OFFICER
    TRUE
    Weaver Jacki
    1.00
    TRUE
    FALSE
    01/01/2010 00:00
    Sheet: Sheet1
    Attached Files Attached Files
    Last edited by AliGW; 06-24-2020 at 04:46 AM.
    Ali


    Enthusiastic self-taught user of MS Excel who's always learning!
    Don't forget to say "thank you" in your thread to anyone who has offered you help.
    You can reward them by clicking on * Add Reputation below their user name on the left, if you wish.

    Forum Rules (updated August 2023): please read them here.

  9. #9
    Forum Moderator AliGW's Avatar
    Join Date
    08-10-2013
    Location
    Retired in Ipswich, Suffolk, but grew up in Sawley, Derbyshire (England)
    MS-Off Ver
    MS 365 Subscription Insider Beta Channel v. 2404 (Windows 11 22H2 64-bit)
    Posts
    80,460

    Re: Conditional remove duplicates based on another column in power query related to (power

    Just wondering if you looked at this when you came back earlier? If it has resolved your issue, please mark the thread as solved.

  10. #10
    Registered User
    Join Date
    08-29-2018
    Location
    Melbourne, Australia
    MS-Off Ver
    2016/365 both
    Posts
    22

    Re: Conditional remove duplicates based on another column in power query related to (power

    Thanks Ali
    Sorry I didn't get back to you last night, but kept getting a HTTP 500 error within the site.

    Yes, it is exactly what I need to go ahead with a distinct table for Power BI fun.

    Thank you to you and the forum moderators who are real experts. I'll now mark the thread closed.

  11. #11
    Forum Moderator AliGW's Avatar
    Join Date
    08-10-2013
    Location
    Retired in Ipswich, Suffolk, but grew up in Sawley, Derbyshire (England)
    MS-Off Ver
    MS 365 Subscription Insider Beta Channel v. 2404 (Windows 11 22H2 64-bit)
    Posts
    80,460

    Re: Conditional remove duplicates based on another column in power query related to (power

    Thanks for letting us know. Being moderators doesn’t make us experts, but Alan and I are getting quite good with PowerQyery.

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Need a solution for power query related work
    By jones143 in forum Excel General
    Replies: 0
    Last Post: 05-16-2020, 02:32 PM
  2. Power query merge (many to many relationship) and remove duplicates
    By richardbreakey in forum Excel Formulas & Functions
    Replies: 6
    Last Post: 05-14-2020, 06:07 PM
  3. Replies: 4
    Last Post: 02-17-2020, 06:03 AM
  4. Power Query - Remove Specific Duplicate Rows (Based on sort criteria)
    By kersplash in forum Excel Formulas & Functions
    Replies: 6
    Last Post: 03-22-2019, 03:30 AM
  5. Replies: 3
    Last Post: 02-19-2019, 10:55 AM
  6. [SOLVED] Power Pivot does not load new column added in Power query
    By ibuhary in forum Excel General
    Replies: 12
    Last Post: 02-19-2019, 03:53 AM
  7. Sub-Forum for Excel Power Tools (Power Query, Power Pivot & Power BI)
    By chullan88 in forum Suggestions for Improvement
    Replies: 10
    Last Post: 06-28-2018, 02:25 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1