Crawling list of web pages for specific value(s) on page

Anyone able to point me in the right direction? My VB skills are null at best, but I'm looking to supply a column of URLs and crawl them for a specific value in the pages crawled and return that in the next column. It seems simple enough in my mind, but of course I'm not familiar with VB.

Bump. I take it this isn't within anyone's realm of expertise.

Hi an_otter,

So you want to have a bunch of urls in one column then press a button and the vba opens the website and looks for a specific value and paste that value next to the link if it exists?

Some questions
1. Is the value the same for all urls?
2. Is the value somewhere in the text of the page or is it in the html code?

Gesendet von meinem HTC One mit Tapatalk

hi loki,

1. The value should be the same for all URLs, yes, they'll all have the same structure.
2. The value is in the text of the page, yes.

Good if no one comes up with something I will look into it the next days

Gesendet von meinem HTC One mit Tapatalk

I really appreciate it. I may even be able to offer some sort of compensation given our group's need for this.

Hi buddy,

I quickly made something for you you can put the URL in Column A and the Word you want to search in Column B.
If the Word is Found within the Page it writes Found in Column C.

Test it with your URLs and give me Feedback if you need changes.
Note that its case Sensitive can change that too if needed.

Greets
Loki

Hi Kyle,

Cool, thanks for the info i did not know that you can do that without IE.

Greets
Loki

Thanks for the scripts, guys. One snag, though:

The value I'll be looking for isn't consistent, but the page and class location of that value will be, nor will it be in the Excel sheet itself for the same reason. All I can provide is the URLs to crawl.

To give you an example, say I'm looking through:

http://www.website.com/page-1.html
http://www.website.com/page-2.html
http://www.website.com/page-3.html

On each there will be an element that's more or less consistent to this:

<div class="product-count">15 items found for</div>

The value of that div ('# items found for') will change depending on each page's product count, but the div location and class remains consistent. In hindsight I should've clarified this detail originally, I'm sorry.

Bump. Any ideas how I could re-gear your guys' macro to take into account the new detail?

try to append this to Kyle's code:

Please Login or Register  to view this content.

Regards,
Rick

You beautiful man, it works perfectly.

I want to thank all of you. If any of you have PayPals or what have you, send me a message, I'd love to throw you some beer money for the effort. Either way, thanks so much.

Hi Otter,

Try this (this is the dirty way :D):

Please Login or Register  to view this content.

Cheers,
Rick

Thanks, Rick, that works fantastically. Is there a way to add an OR clause to that StrStart variable? As a sort of fall-back element to look for? Like:


StrStart = "<div class=""product-count"">" OR "<div style=""float:left;font-size:12px;color:#52335F;"">"


I know I'm being a picky brat at this point, but that's my last quibble.

God you're good, that works pretty much perfectly. Thanks again, Rick!

Sorry I'm a bit late to this party. I just received an email from ExcelForum about it. While it seems a good solution has been offered (and one that does not involve Internet Explorer), I have a different solution (that does involve Internet Explorer) but does not require VBA skills - tho - if you have VBA skills the provided function library can be used to do far more than just web crawling.

It is far too comprehensive to put into a forum post so I offer this 4 minute demo video so readers can determine if it is something they could use. It is free (I am not a commercial enterprise) to any ExcelForum member who wants it and for the benefit of all, the code is unprotected.

https://youtu.be/mxZBVMfcUKc

Hi All
Anybody can you please tell me how to crawl data/content from a particular web page.