+ Reply to Thread
Results 1 to 5 of 5

Scraping data from website

  1. #1
    Registered User
    Join Date
    12-30-2015
    Location
    florida
    MS-Off Ver
    2010
    Posts
    3

    Scraping data from website

    I am very new to vba programming and am still learning the rules about the language. I am trying to write a macro that scrapes data from a website. The problem that i am having right now is that not all of the data is being scraped. This macro will be run on a daily basis, so the amount of data to be scraped will be different. Basically, for the data that is to be scraped today, there are 7 pages of data, but the macro that i have right now will only scrape until page 6. I cannot seem to figure out why it will only scrape until page 6 and not page 7. When i run this macro for the data from 2 days, it pulls in and scrapes all of the data and there were about 8 pages of data to scrape from 2 days ago. This is the macro that I have so far. Any help is greatly appreciated.

    Please Login or Register  to view this content.
    Last edited by novicevba; 12-30-2015 at 10:58 AM.

  2. #2
    Forum Expert JasperD's Avatar
    Join Date
    05-07-2013
    Location
    Netherlands
    MS-Off Ver
    Excel 2016
    Posts
    1,393

    Re: Scraping data from website

    does this still need your help?

  3. #3
    Registered User
    Join Date
    12-30-2015
    Location
    florida
    MS-Off Ver
    2010
    Posts
    3

    Re: Scraping data from website

    I was able to figure out the solution to the problems that I was having. The code is working perfectly the way that I intended the code to. Thank you

  4. #4
    Forum Expert JasperD's Avatar
    Join Date
    05-07-2013
    Location
    Netherlands
    MS-Off Ver
    Excel 2016
    Posts
    1,393

    Re: Scraping data from website

    Glad to hear!
    In that case, please mark this thread as solved - it would also be good for future reference of people if you can include your working code, in case someone might be looking for it.

  5. #5
    Registered User
    Join Date
    12-30-2015
    Location
    florida
    MS-Off Ver
    2010
    Posts
    3

    Re: Scraping data from website

    Thanks!

    This is the finally working code where I am able to scrape the data in the way I had initially intended on doing so.

    [CODE]
    'Macro to query Daily Activity Search for DFB Counties
    'Run Monday to pull data from Friday


    Sub queryActivityDailyMforFWorking()
    On Error GoTo Err_queryActivityDailyMforFWorking

    Const RowsPerPage As Byte = 20
    Const DeleteHeader As Boolean = True

    Dim nextrow As Integer, maxrow As Integer, i As Integer
    Dim dates As Date

    dates = Date - 3

    Application.ScreenUpdating = False
    Application.DisplayStatusBar = True

    nextrow = 1
    For i = 1 To 50
    Application.StatusBar = "Processing Page " & i
    With ActiveSheet.QueryTables.Add(Connection:= _
    "URL;https://www.myfloridalicense.com/delinquency_results.asp?SID=&page=" & i & "&county_1=16&county_1=21&county_1=23&county_1=32&county_1=36&county_1=41&county_1=46&county_1=53&county_1=54&county_1=57&county_1=60&county_1=66&status=R&send_date=" & Format(dates, "m/d/yyyy") & "&search_1.x=1", _
    Destination:=Range("A" & nextrow))
    '.Name = _
    "2015&search_1.x=40&search_1.y=11&date=on&county_1=AL&lic_num_del=&lic_num_rep=&status=NS&biz_name=&owner_name="
    .FieldNames = False
    .RowNumbers = False
    .FillAdjacentFormulas = False
    .PreserveFormatting = True
    .RefreshOnFileOpen = False
    .BackgroundQuery = True
    .RefreshStyle = xlInsertDeleteCells
    .SavePassword = False
    .SaveData = True
    .AdjustColumnWidth = True
    .RefreshPeriod = 0
    .WebSelectionType = xlSpecifiedTables
    .WebFormatting = xlWebFormattingNone
    .WebTables = "10"
    .WebPreFormattedTextToColumns = True
    .WebConsecutiveDelimitersAsOne = True
    .WebSingleBlockTextImport = False
    .WebDisableDateRecognition = False
    .WebDisableRedirections = False
    .Refresh BackgroundQuery:=False
    End With

    ' Delete the header as required
    If DeleteHeader And i > 1 And ActiveSheet.Cells(nextrow, 1).Value = "License" Then ActiveSheet.Cells(nextrow, 1).EntireRow.Delete

    ' Find the bottom row
    maxrow = Application.WorksheetFunction.Max(ActiveSheet.Cells(Rows.Count, 1).End(xlUp).row, ActiveSheet.Cells(Rows.Count, 2).End(xlUp).row)
    ' Stop scraping if a full page wasn't returned
    If (maxrow - nextrow) < (RowsPerPage - IIf(DeleteHeader, 1, 0)) Then
    Exit For
    ' Otherwise set the row for the next page of data
    Else
    nextrow = maxrow + 1
    End If
    Next i

    Application.StatusBar = "Formatting data"

    'autofit columns
    ActiveSheet.Columns.EntireColumn.AutoFit

    'check for filter, if not then turn on filter
    ActiveSheet.AutoFilterMode = False
    If Not ActiveSheet.AutoFilterMode Then ActiveSheet.Range("A:G").AutoFilter

    'Align text left
    With ActiveSheet.Cells
    .HorizontalAlignment = xlLeft
    .VerticalAlignment = xlBottom
    .WrapText = False
    .Orientation = 0
    .AddIndent = False
    .IndentLevel = 0
    .ShrinkToFit = False
    .ReadingOrder = xlContext
    .MergeCells = False
    End With

    Exit_queryActivityDailyMforFWorking:
    Application.StatusBar = False
    Application.ScreenUpdating = True
    Exit Sub

    Err_queryActivityDailyMforFWorking:
    MsgBox Err.Description, vbCritical + vbOKOnly, Err.Number & " - Web Scraping Error"
    Resume Exit_queryActivityDailyMforFWorking

    End Sub
    [CODE]

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Web Scraping Data From A Website Where Hyperlink Does Not Change
    By NeedForExcel in forum Excel Programming / VBA / Macros
    Replies: 4
    Last Post: 10-01-2015, 02:27 AM
  2. Data Scraping from the WebSite
    By sathis in forum Excel Programming / VBA / Macros
    Replies: 11
    Last Post: 07-26-2015, 10:46 AM
  3. problem while scraping data from website
    By dps700 in forum Excel Programming / VBA / Macros
    Replies: 0
    Last Post: 12-09-2014, 09:46 AM
  4. Automated data scraping from website into excel
    By kavin in forum Excel Programming / VBA / Macros
    Replies: 1
    Last Post: 09-10-2014, 01:42 PM
  5. Replies: 1
    Last Post: 02-17-2014, 11:21 PM
  6. scraping data from a website
    By redpanda in forum Excel Programming / VBA / Macros
    Replies: 9
    Last Post: 07-05-2012, 01:20 PM
  7. Scraping website data into Excel
    By oliver79 in forum Excel General
    Replies: 0
    Last Post: 06-13-2010, 02:15 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1