+ Reply to Thread
Results 1 to 19 of 19

Mass data transfer

  1. #1
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10

    Mass data transfer

    Before I start talking about my excel needs I just want to wish everyone a merry x mas!!

    Now lets talk about excel, I need to transfer a large amount data (~2,5 million lines of data) from one spreadsheet to another while making sure the destination file is kept in a certain format.

    Spreadsheet "oxno" is the file that needs filling out with data provided to me in spreadsheet "ox"
    The value that is the matching criteria will be the ITS account number (Field B in "ox" and field A in "oxno") and the currency (Field B in "oxno" and field G in "ox")
    So I want a macro to look in "oxno" and use field the values of field "a" & "b" and search for the them in "ox" in fields "B" & "G". Once it finds a match it should copy the value of field "I" in "ox" and paste into field "n" on "oxno".
    The next data to be copied should be what goes into field "O" in "oxno" this data is derived from field "L" or "M" but only when its of numerical value with one non numerical field in it (usually a "/").
    Last value that I need copied over is Field "T" on "oxno" this is derived from field "J" or "M".

    I realise the potential conflict because two sets of queries will be looking to gather values from field "M" on "ox". That is why I am hoping when we look for value for field "O" in "oxno" we only look for numbers with one non numerical field in it (usually a "/") .

    After finishing writing this up, I realise how wordy this has become appologies for that, but hopefully someone will be able to understand what I am looking to achieve and will be able to suggest a solution.

    Thanks in advance.

    Nowshad
    Attached Files Attached Files
    Last edited by Nowsh; 12-27-2008 at 08:39 PM.

  2. #2
    Forum Expert shg's Avatar
    Join Date
    06-20-2007
    Location
    The Great State of Texas
    MS-Off Ver
    2003, 2010
    Posts
    40,678
    You're aware that no version of Excel accommodates more than about a million rows?

    Maybe Access ...
    Entia non sunt multiplicanda sine necessitate

  3. #3
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10
    I am aware of that, I was just stating the total lines of data is 2.5 M lines, not in one spreadsheet, but spread accross many. The original data as you correctly stated is in Access and is a huge 800 MB big

  4. #4
    Forum Expert
    Join Date
    01-15-2007
    Location
    Brisbane, Australia
    MS-Off Ver
    2007
    Posts
    6,591
    Hi

    Using your 2 files, put the macro below into a general module in oxno.xls. Have both spreadsheets open and run. Check the output and see how it goes.

    Please Login or Register  to view this content.
    rylo

  5. #5
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10

    Talking

    Wow rylo thanks a lot for that, you are a real legend!

    Field "N" is being generated perfectly. Field "O" is copied over correctly more than not, but no data at all is being copied over for "T"

    Nowshad

  6. #6
    Forum Expert
    Join Date
    01-15-2007
    Location
    Brisbane, Australia
    MS-Off Ver
    2007
    Posts
    6,591
    Hi

    Using the example file that we have been using, can you give instances of
    1) Field O that is incorrect, what it should be and why.
    2) Examples of Column T where it should be bringing back a result.

    rylo

  7. #7
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10
    Duh, stupid me, please find attached the outcome.

    I will focus on the first set of results (line 4-17 on spreadsheet "oxno" & lne 2-79 on spreadsheet "ox") and fill the fields in light yellow where nothing was copied over to field "O" and pale blue for field "T".
    Also there was one instance where the field "N" was not copied over, I have put that in rose.

    Thanks again for all your help it is really appreciated!
    Attached Files Attached Files

  8. #8
    Forum Expert
    Join Date
    01-15-2007
    Location
    Brisbane, Australia
    MS-Off Ver
    2007
    Posts
    6,591
    Hi

    Looking quickly at some of the results in oxno.xls!O, you have entries that have 2 non numeric characters, yet your requirement was
    its of numerical value with one non numerical field in it (usually a "/").
    . Can you please clarify.

    Also, I applied the same logic to the data for column T. Does this not apply?

    For the same block in rows 4-17, can you please explain for each and every one WHY the item was selected, where it came from (row number) and the logic that selected that item. Update the example files and put the reasons etc into the next columns (P and U).

    rylo

  9. #9
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10
    Sorry you are correct I did state originally I only needed data to be transfered that was numerical with max of 1 non numerical character, can I correct myself and request 5 non-numerical charcters?
    Again I did not clearly state the requirments for Field "T" so appologies for that. This field will have similar data as field "N" so the requirements for field "T" should be the same as field "N".

    Thanks

  10. #10
    Forum Expert
    Join Date
    01-15-2007
    Location
    Brisbane, Australia
    MS-Off Ver
    2007
    Posts
    6,591
    Hi

    When you say 5, do you mean that it must have 5 non numeric, or up to and including 5 non numeric (excluding spaces).

    rylo

  11. #11
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10
    Hello

    I mean up to 5 (excluding spaces), so if the data found by macro has more than 5 non numerical numbers it can ignore it and not copy it over.

    Thanks

  12. #12
    Forum Expert
    Join Date
    01-15-2007
    Location
    Brisbane, Australia
    MS-Off Ver
    2007
    Posts
    6,591
    Hi

    the entry in row 10 of oxno.xls (012292041, NLG) has 3 matches in ox.xls (rows 30, 72, 78). How is the correct result determined? And are there likely to be many that have multiple possible answers? If so, is there a unique combination that can be used?

    rylo

  13. #13
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10
    If it is possible to specifiy that the first match should be copied over for field "N" from "I". If that is do-able that would be the best solution, because the data will always be same just in different lines.

  14. #14
    Forum Expert
    Join Date
    01-15-2007
    Location
    Brisbane, Australia
    MS-Off Ver
    2007
    Posts
    6,591
    Hi

    For the column T output, if it is the same selection as column N, then how do you determine which entry to take from columns J and M? J if it is non blank otherwise M regardless of what is in it?

    As I requested earlier, can you please complete each of the items in rows 4 - 17, explain where the data came from and WHY that result was selected.



    rylo

  15. #15
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10
    Hi

    Quote Originally Posted by rylo View Post
    Hi

    For the column T output, if it is the same selection as column N, then how do you determine which entry to take from columns J and M? J if it is non blank otherwise M regardless of what is in it?

    rylo
    Copying value for "T" from "J" if non blank otherwise "M" as long as "M" is no more than 2 numerical characters, would be the perfect solution.

    Also I am attaching "oxno" with comments in terms of where the data has come from and what criteria is attached to that.

    Thanks
    Attached Files Attached Files

  16. #16
    Forum Expert
    Join Date
    01-15-2007
    Location
    Brisbane, Australia
    MS-Off Ver
    2007
    Posts
    6,591
    Hi

    OK, here goes again.

    One thing I've noticed with oxno.xls, is that you have an ID in the heading row for most of the entries except row 2. You will have to correct columns A and B to make sure the headings are in those columns.

    If this brings back some incorrect entries, note which ones, and explain where it should come from and why....

    rylo

    Please Login or Register  to view this content.

  17. #17
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10

    Talking

    Oh my god Rylo, you are a legend! Do you guys collect donations or something, because you helping me with this deserves one

  18. #18
    Forum Expert
    Join Date
    01-15-2007
    Location
    Brisbane, Australia
    MS-Off Ver
    2007
    Posts
    6,591
    Hi

    I gather that this has solved your request (Gees I sure hope so). If this is right, can you please go back to your original post and mark it as solved.

    And don't forget to recommend the site and tell everyone just how helpful the site has been......


    rylo

  19. #19
    Registered User
    Join Date
    12-24-2008
    Location
    London
    Posts
    10

    Talking

    haha indeed it has, thanks again, and dont worry about me telling people about this site, you guys (especially you Rylo) are amazing!!

    I shall mark this thread as solved.

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1