+ Reply to Thread
Results 1 to 12 of 12

DIFFICULT ONE: Scraping data from web when it's not in the HTML

  1. #1
    Registered User
    Join Date
    12-26-2017
    Location
    London, England
    MS-Off Ver
    2016
    Posts
    4

    DIFFICULT ONE: Scraping data from web when it's not in the HTML

    Hey y'all!

    I was wondering if it was possible to pull data from the tables of the webpage below. The itchy point is that the HTML code of the website doesn't contain the info of the tables, which makes us think of a different solution than a simple web query as we've been always doing.


    So, my expert friends, what are your proposals? I have no clue right now and it is a really interesting and useful topic for those who want to deep in web scrapping!

    Thaaank you, link below

    http://quote.morningstar.ca/Quicktak...n-CA&ops=clear

  2. #2
    Forum Expert
    Join Date
    03-28-2012
    Location
    TBA
    MS-Off Ver
    Office 365
    Posts
    12,454

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    The data are generated by JavaScript. There are not many tools at your disposal as far as VBA goes. You could try Selenium- third part tool( You need to install it). You get a better luck with Python, since it has many API's.

  3. #3
    Forum Guru Kyle123's Avatar
    Join Date
    03-10-2010
    Location
    Leeds
    MS-Off Ver
    365 Win 11
    Posts
    7,238

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    Starter for 10, needs making more robust and tweaking to get what you need:
    Please Login or Register  to view this content.

  4. #4
    Forum Guru Norie's Avatar
    Join Date
    02-02-2005
    Location
    Stirling, Scotland
    MS-Off Ver
    Microsoft Office 365
    Posts
    19,643

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    As far as I can see the data is available from that page is in the HTML.

    What data are you trying to get?
    If posting code please use code tags, see here.

  5. #5
    Forum Expert
    Join Date
    03-28-2012
    Location
    TBA
    MS-Off Ver
    Office 365
    Posts
    12,454

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    Hi Norie,
    If you run the URL in "Get & Transform" from web function and if the page has a table, you see a table. That is what the OP probably did. Excel 2016 "Get & Transform" addition has a more robust way of getting data from a table than the old import from a web. When I tried it, I could not see any table and hence for my wrong assumption.
    Kyle knows how to read the pages and find the right one.

  6. #6
    Forum Guru Norie's Avatar
    Join Date
    02-02-2005
    Location
    Stirling, Scotland
    MS-Off Ver
    Microsoft Office 365
    Posts
    19,643

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    AB33

    Why would I do that?

    I opened the page and the used the browser tools to have a look at the HTML behind it.

  7. #7
    Forum Expert
    Join Date
    03-28-2012
    Location
    TBA
    MS-Off Ver
    Office 365
    Posts
    12,454

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    Hi Norie,
    The OP's URL- http://quote.morningstar.ca/Quicktak...n-CA&ops=clear - does not have a table. If you view the source page, there is not a word called "Table". Yes, if you inspect the individual elements, you can see there is a table, but you would not be able to scrap any data unless you go in to different pages of the site. That is what Kyle did.
    Last edited by AB33; 12-28-2017 at 06:00 PM.

  8. #8
    Forum Guru Norie's Avatar
    Join Date
    02-02-2005
    Location
    Stirling, Scotland
    MS-Off Ver
    Microsoft Office 365
    Posts
    19,643

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    AB33

    I'm kind of confused - when I look at the HTML behind the page I'm definitely seeing 'table', 'tr' etc elememts.

  9. #9
    Registered User
    Join Date
    12-26-2017
    Location
    London, England
    MS-Off Ver
    2016
    Posts
    4

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    Quote Originally Posted by Kyle123 View Post
    Starter for 10, needs making more robust and tweaking to get what you need:
    Please Login or Register  to view this content.
    Hey friend! This is just FREAKING AMAZING!!! I almost understand everything but... how did you get the link where you pulled all the data from? it is not the same and I am not able to figure out where does it come from!


    beast!!!

  10. #10
    Forum Guru Kyle123's Avatar
    Join Date
    03-10-2010
    Location
    Leeds
    MS-Off Ver
    365 Win 11
    Posts
    7,238

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    You need to work through the network requests that the page makes using your browser developer tools and figure out which one pulls the data in

  11. #11
    Registered User
    Join Date
    12-26-2017
    Location
    London, England
    MS-Off Ver
    2016
    Posts
    4
    Quote Originally Posted by Kyle123 View Post
    You need to work through the network requests that the page makes using your browser developer tools and figure out which one pulls the data in
    Yeah I understand but as I am not an expert, could you explain how did you exactly got it in this particular case? If it is not tedious, it would be really interesting! Thank you for your savage solution

  12. #12
    Forum Guru Kyle123's Avatar
    Join Date
    03-10-2010
    Location
    Leeds
    MS-Off Ver
    365 Win 11
    Posts
    7,238

    Re: DIFFICULT ONE: Scrapping data from web when it's not in the HTML

    I'm afraid it is tedious and rather in depth, it's probably enough to fill a couple of blog articles. Do you know anything about web development (HTTP, JavaScript, AJAX & JSON in this case)? If you do it'll be much quicker to explain

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. HTML scraping with a help of Selenium
    By ChipsSlave in forum Excel Programming / VBA / Macros
    Replies: 1
    Last Post: 05-31-2018, 12:12 AM
  2. VBA scraping of HTML
    By ChipsSlave in forum Excel Programming / VBA / Macros
    Replies: 6
    Last Post: 07-28-2017, 01:06 PM
  3. [SOLVED] VBA scraping specific value from website HTML
    By Strutland in forum Excel Programming / VBA / Macros
    Replies: 3
    Last Post: 03-25-2017, 10:06 AM
  4. [SOLVED] Data Scraping
    By underwater24 in forum Excel Programming / VBA / Macros
    Replies: 3
    Last Post: 10-30-2016, 11:36 PM
  5. Replies: 2
    Last Post: 01-07-2016, 12:25 PM
  6. [SOLVED] vlookup on an html table; difficult
    By perducci in forum Excel Programming / VBA / Macros
    Replies: 2
    Last Post: 12-18-2012, 01:38 PM
  7. HTML scraping using VBA
    By MacroLide in forum Excel Programming / VBA / Macros
    Replies: 4
    Last Post: 05-26-2012, 11:12 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1