Web scraping problem

Hi guys I have run into a little problem as one of the sites I usally scrape info from has just done a complete overhaul. As of now they use javascript to generate their content.
So I had to rebuild my script. Im almost finished but Im missing one element from the site and I can not figure out how to scrape it. html code is here:

Please Login or Register  to view this content.

As of now there are 243 elements like this. I need only this data-league-name node
I have tried something like this:

Please Login or Register  to view this content.

But this does not return everything from the above code only this:

Please Login or Register  to view this content.

Im not sure how to I should phrase my code to get the league from this site :/ But I know It should be possible as all of this is stored in the .DocumentElement.innerHTML

Any help making me handle this would be much appreciated.

Frederik

Have you tried using an id element? Id is usually unique per page and returns a single value.

No I have not tried that . Do you mean like this:

id="1426760"

I can try this now, but how would I figure out the id without checking manually ?

No I have not tried that . Do you mean like this:

id="1426760"

I can try this now, but how would I figure out the id without checking manually ?

Here is how it looks like in the explorer console:

Please Login or Register  to view this content.

X = HTMLDoc.GetElementByID("1426760").innertext

Originally Posted by Sean Thomas

As you have already got a collection of all the elements with the classname
Set HTML_League = HTMLDoc.getElementsByClassName("match_line score_row other_match o_true")
Then you need to loop through these elements to find the one that has the id that you want

you could do something like this

Please Login or Register  to view this content.

Problem is that when I extract this element HTMLDoc.getElementsByClassName("match_line score_row other_match o_true")
The data-league-name node is not found in it hmmm...

data-note data-country-name="WORLD (FIFA)"
Is not an inner text, but rather most likely is an attribute.
You could try innerHTML not innertext and see what it returns.

X = HTMLDoc.getElementById("1426760").innerHTML

Originally Posted by AB33

data-note data-country-name="WORLD (FIFA)"
Is not an inner text, but rather most likely is an attribute.
You could try innerHTML not innertext and see what it returns.

X = HTMLDoc.getElementById("1426760").innerHTML

Yes that is what I also figured however with my skillset its not so easy Ill try , However I need to figure out how to get all of these attributes without searching by id name as i have no idea how to get the ID number..

<div id="1326133" class="match_line score_row live_match o_true" data-statustype="live" data-ko="19:00" data-home-team="GUINEA" data-away-team="TUNISIA" data-league-sort="52" data-correction="-7" data-matchday="2017-10-07" data-game-status="2 HF" data-league-code="39384" data-league-name="WORLD CUP RUSSIA 2018" data-note data-country-name="WORLD (FIFA)" data-league-round="MATCHDAY 5" data-league-short="WCQ" data-home-id="10121" data-away-id="20360">…</div>

class="match_line score_row live_match o_true"
is the class CSS with "match_line score_row live_match o_true" attribute.
Since class can be applied to many elements, it is considered as a collection, while id id="1326133" is unique to each element.
Since you are using IE, you can also find the attribute value.

Originally Posted by AB33

<div id="1326133" class="match_line score_row live_match o_true" data-statustype="live" data-ko="19:00" data-home-team="GUINEA" data-away-team="TUNISIA" data-league-sort="52" data-correction="-7" data-matchday="2017-10-07" data-game-status="2 HF" data-league-code="39384" data-league-name="WORLD CUP RUSSIA 2018" data-note data-country-name="WORLD (FIFA)" data-league-round="MATCHDAY 5" data-league-short="WCQ" data-home-id="10121" data-away-id="20360">…</div>

class="match_line score_row live_match o_true"
is the class CSS with "match_line score_row live_match o_true" attribute.
Since class can be applied to many elements, it is considered as a collection, while id id="1326133" is unique to each element.
Since you are using IE, you can also find the attribute value.

im not sure I understand what you mean.. ?

I mean you can not create your ID or class. I thought you mentioned you do not know how to get an ID.
In the above div element, you have both ID and Class to access the div element.

Is it secure website? If not, could you attach the URL?

Ok, let me have a go. Which data do you want to extract?

thanks so much its really only the league name I need (not the short name) as all the others I have a handle on

"Data-league-name"

Let's say

<div id="1484267" class="match_line score_row live_match o_true" data-statustype="live" data-ko="14:00" data-home-team="YEOVIL (W)" data-away-team="SUNDERLAND (W)" data-league-sort="901" data-correction="0" data-matchday="2017-10-08" data-game-status="2 HF" data-league-code="42784" data-league-name="WOMENS SUPER LEAGUE" data-note="Venue: Huish Park. Turf: Natural. Capacity: 9,565." data-country-name="ENGLAND" data-league-round="3" data-league-short="WSL" data-home-id="50043" data-away-id="50027">

Is it right you want to extract

data-league-name="WOMENS SUPER LEAGUE"

"WOMENS SUPER LEAGUE"
from
data-league-name?

That is correct. but I would need a way to extract all the leagues

So far I have All countries, Team A , Team B and Starting times, Only league is missing, I was able to extract the short league name but that does not work with my overall setup.

You are now confusing me!
There are 17 "match_line score_row live_match o_true"
and I assume all the names are found within this div.
Which one have you already extracted?

when I hit refresh now I recived 326 rows of data

In col B I have: HTML_Time = HTMLDoc.getElementsByClassName("score_ko score_cell")
Col C : HTML_Country = HTMLDoc.getElementsByClassName("tooltip_flag")
Col D: Should be league , this is what I need help with
Col E: HTML_TeamA = HTMLDoc.getElementsByClassName("score_home_txt score_cell wrap")
Col F: HTML_TeamB = HTMLDoc.getElementsByClassName("score_away_txt score_cell wrap")

basically I want to extract it looking quite similar to their site

No, I do not think I am getting what you are trying to do.
For e.g
if I extract All the div element, I get

Set HTML_TeamA = HTMLDoc.getElementsByClassName("match_line score_row live_match o_true")

"
16:00

FIN





SHOW GAMES FROM ITALY

B



PERUGIA


4

1

1


PRO VERCELLI


17

2




0

1


1

5

"


Is one of them and SHOW GAMES FROM ITALY is found.
is not SHOW GAMES FROM ITALY what you are after?

Er, I think I have came across this before. I think the data are generated by JavaScript. Despite the data appears to be on the table, but it is not. Your best bet is to get an API if the site owner has one. I suspect the owner is blocking web scrapping. They let you to use IE as it is not stable and reliable.
Sorry, I can not more helpful.

Wow thanks buddy. This looks interesting , I can see that you used some time to compile this. Can you confirm this is working? , I see that you have an http request for this , but when I tried with an http request the javascript part of the page does not get generated ?

As of now I'm off to work but will check this out when I'm back . But this looks real promising )

Frederik

Thanks for your help with this, I'm little intressted in how you were able to get the generated javascipt content without ie.. Maybe this would work on other sites with js also?

I'm currently away from the system but I'll test as soon as I'm back home

Frederik

You never Need to use Internet explorer since you never really need to execute JavaScript. You just read the data from the same source as the JavaScript does, in this case the data is in the HTML - in html attributes, Sean's code simply reads the values of the html attributes.

Ok so I just came home from work , Been looking forward to test this all day

Ive tested:

GetWebData()

which returned:
score_away_txt score_cell wrap
score_home_txt score_cell wrap
score_ko score_cell
score_league_flag score_cell
score_league_txt score_cell
scoreh_ht score_cell centerTXT
scorea_ht score_cell centerTXT
scoreh_ft score_cell centerTXT
scorea_ft score_cell centerTXT

Also tested

GetinnerHTML()
But that did not return anything ?

Maybe im doing something wrong ?

Frederik

Ill try in a new workbook. I have actually found a solution for my problem at this point by saving the htmldoc and just searching trough the txt file for the league names . But I'm really interested in your way to tackle this problem. Especially after hearing its possible to get javascript content using your technique. I really thought I would need selenium wrapper for this ? As I have a few tricky sites I really want to scrape the content of. But those sites are more and less 100% javascript and Flash generated and IE can not handle these type of site. When I have tried downloading the htmldoc at these type of sites It comes out almost empty...

Yes, You are better off with third-parties applications, particularly if you do lots of data crawling and scrapping.
Python has many more utilities and functions for getting data from dynamic sites. In fact, Selenium is written in Python.

I was thinking about taking all of my work out of excel , but that's a gigantic task...

Yes, that is why many people are stack with excel, but VBA has limited functionalities. Many of the Third-parties have the tool to export data in to CSV and excel formats. In some aspects it works much better and faster to open a CSV file. You are also not constrained with 1m rows of excel.

Yes you are right, I have started doing some research of which coding language to choose should I choose to move my project out of excel , but i'm stuck at the selection , lol. It was recomended to me to try Visual basic as it is quite similar to excel vba. But I've also heard quite a lot about Python as well. I have 0 skills with either of them , but I think Im a fast learner and I spend quite some time on something when Im intrested in it. In your opinion which code lang. is the way to go, or does it even matter ?

Python will be less of a culture shock and you'll find more tutorials out there on web scraping

OK guys thanks alot for all the info , This thread has been very informative for me. I think I will look into python..

cheers

Whilst VBA is an offshoot of VB6, VB.NET is a completely different animal. Despite the rather confusing naming, it's actually nothing like VB6 - it's primarily object orientated for starters which VB6 never was. Most of the classic VB developers I know, migrated to VB.NET, hated it and moved over to C#, the issue is that it sort of smells like VB6 but is completely different under the skin, so much so they got frustrated and found it easier to get their heads around a completely different syntax (C#) - C# and VB.NET compile to the same language anyway.

To learn VB.NET you need to learn OOP, for Python you don't. It's a scripting language that lets you write in the same procedural manner you'll be using in Excel, it's easier to write scripts that focus on getting things done. It's also open source, meaning that it's widely used (much more so than a Microsoft stack - which is predominantly corporate focused) and so there are loads of tutorials and 3rd party libraries to help you - python's also widely used for webs scraping so there's plenty of resource out there.

Personally, I don't like python, but it's a great getting started language that's easier to get started with and doesn't force you to learn architecture and OOP concepts first (which you need to really with something like VB, C#, Java etc). I'm with AB33 on this one, Python would be a good choice, but I don't think that Excel is particularly bad either.

Ok then after hearing all of this I will definitely go for python. can I build a 100% program with this for windows? where can I download a starter pack ?

and you are right excel is not a bad at all , but would be sweet to get all of my work out of it.