+ Reply to Thread
Results 1 to 15 of 15

what kind of parameters should be sent to web server?

  1. #1
    Forum Contributor
    Join Date
    11-12-2010
    Location
    HK
    MS-Off Ver
    Excel 2010
    Posts
    189

    what kind of parameters should be sent to web server?

    Hi,

    I got help from one of this forum contributors a month ago to send info to the web server. The red code is the information that is required by a web page in order to proceed to the next page.
    Please Login or Register  to view this content.
    However, when I change to below website, I have no ideas of what should be included or should not be included from HTML source after .send.
    HTML Code: 
    Could anyone tell me the principle of picking the necessary elements? Thanks
    Last edited by lubbamkt; 11-01-2014 at 02:42 AM.
    English is not my first language

  2. #2
    Forum Contributor
    Join Date
    11-12-2010
    Location
    HK
    MS-Off Ver
    Excel 2010
    Posts
    189

    Re: what kind of element should be sent to web server?

    Can anyone give me a hand please?

  3. #3
    Forum Guru Kyle123's Avatar
    Join Date
    03-10-2010
    Location
    Leeds
    MS-Off Ver
    365 Win 11
    Posts
    7,238

    Re: what kind of element should be sent to web server?

    Inspect the traffic, open up dev tools in your browser, go to the network tab and have a look what is in the Request body

  4. #4
    Forum Contributor
    Join Date
    11-12-2010
    Location
    HK
    MS-Off Ver
    Excel 2010
    Posts
    189

    Re: what kind of element should be sent to web server?

    do you mean to check the html source?
    if so, i did check them, but cannot identify which elements must be entered in order to proceed to next page.

  5. #5
    Forum Guru Kyle123's Avatar
    Join Date
    03-10-2010
    Location
    Leeds
    MS-Off Ver
    365 Win 11
    Posts
    7,238

    Re: what kind of element should be sent to web server?

    No I don't, I mean the network panel in dev tools

  6. #6
    Forum Contributor
    Join Date
    11-12-2010
    Location
    HK
    MS-Off Ver
    Excel 2010
    Posts
    189

    Re: what kind of element should be sent to web server?

    Can someone give me an advice on below code, I cannot proceed to next page successfully. I guess there must be something missed in the red code.
    Please Login or Register  to view this content.

  7. #7
    Valued Forum Contributor xlbiznes's Avatar
    Join Date
    02-22-2013
    Location
    Bahrain
    MS-Off Ver
    Excel 2007
    Posts
    1,223

    Re: what kind of element should be sent to web server?

    Hi,

    try this :

    Please Login or Register  to view this content.
    Happy Computing ,

    Xlbiznes.

    To show your appreciation please click *

  8. #8
    Valued Forum Contributor xlbiznes's Avatar
    Join Date
    02-22-2013
    Location
    Bahrain
    MS-Off Ver
    Excel 2007
    Posts
    1,223

    Re: what kind of element should be sent to web server?

    Check this sample workbook that extracts the elements from a html document.

  9. #9
    Forum Guru Kyle123's Avatar
    Join Date
    03-10-2010
    Location
    Leeds
    MS-Off Ver
    365 Win 11
    Posts
    7,238

    Re: what kind of element should be sent to web server?

    @xlbiznes, you're much better off using WinHTTP over MSXML since it has more options and will handle cookies properly for you. Also, your code will only work for GET requests. The site the OP is trying to scrape from uses .Net webforms (which are pretty horrible) and use POST predominantly.

    @lubbamkt, you haven't sent all the request. You also need to send the VIEWSTATE and __EVENTVALIDATION inputs

  10. #10
    Forum Contributor
    Join Date
    11-12-2010
    Location
    HK
    MS-Off Ver
    Excel 2010
    Posts
    189

    Re: what kind of element should be sent to web server?

    Quote Originally Posted by Kyle123 View Post
    @xlbiznes, you're much better off using WinHTTP over MSXML since it has more options and will handle cookies properly for you. Also, your code will only work for GET requests. The site the OP is trying to scrape from uses .Net webforms (which are pretty horrible) and use POST predominantly.

    @lubbamkt, you haven't sent all the request. You also need to send the VIEWSTATE and __EVENTVALIDATION inputs
    Thanks for your advice, I tried to add VIEWSTATE and __EVENTVALIDATION inputs, but I don't know why I cannot set them in the code, please see below in red.
    And should I use GET or POST in this case? Am I right to use the green code if I use WinHTTP?
    Thank you so much
    Please Login or Register  to view this content.

  11. #11
    Forum Contributor
    Join Date
    11-12-2010
    Location
    HK
    MS-Off Ver
    Excel 2010
    Posts
    189

    Re: what kind of element should be sent to web server?

    Quote Originally Posted by xlbiznes View Post
    Check this sample workbook that extracts the elements from a html document.
    Thanks xlbiznes, I entered http://sdinotice.hkex.com.hk/di/NSSr...&in=1&lang=EN& into URL_LIST sheet, but nothing return, is there anything I went wrong?

  12. #12
    Forum Guru Kyle123's Avatar
    Join Date
    03-10-2010
    Location
    Leeds
    MS-Off Ver
    365 Win 11
    Posts
    7,238

    Re: what kind of element should be sent to web server?

    No, your green code is wrong, it was right in your first post.

    Also, why are you now putting the parameters on the end of the url? These go in the body like you were originally - ignore all the code by xlbiznes, it's completely wrong - you now have all the params you need, just put them in your original code

  13. #13
    Forum Contributor
    Join Date
    11-12-2010
    Location
    HK
    MS-Off Ver
    Excel 2010
    Posts
    189

    Re: what kind of element should be sent to web server?

    Quote Originally Posted by Kyle123 View Post
    No, your green code is wrong, it was right in your first post.

    Also, why are you now putting the parameters on the end of the url? These go in the body like you were originally - ignore all the code by xlbiznes, it's completely wrong - you now have all the params you need, just put them in your original code
    Hi Kyle123, do you mean like this?
    Please Login or Register  to view this content.
    I got a difficulty in setting __VIEWSTATE and __EVENTVALIDATION. Because of the length of their value, I tried to set their value into VS and EV as shown in above code, but failed.

    I copy the tab of __VIEWSTATE from the web for your reference.
    Please Login or Register  to view this content.
    Please help, I am dealing with this web page for days without success.
    Last edited by lubbamkt; 10-31-2014 at 02:00 PM.

  14. #14
    Valued Forum Contributor xlbiznes's Avatar
    Join Date
    02-22-2013
    Location
    Bahrain
    MS-Off Ver
    Excel 2007
    Posts
    1,223

    Re: what kind of element should be sent to web server?

    hi lubbamkt,

    try this file.

  15. #15
    Forum Contributor
    Join Date
    11-12-2010
    Location
    HK
    MS-Off Ver
    Excel 2010
    Posts
    189

    Re: what kind of element should be sent to web server?

    Quote Originally Posted by lubbamkt View Post
    Hi Kyle123, do you mean like this?
    Please Login or Register  to view this content.
    I got a difficulty in setting __VIEWSTATE and __EVENTVALIDATION. Because of the length of their value, I tried to set their value into VS and EV as shown in above code, but failed.

    I copy the tab of __VIEWSTATE from the web for your reference.
    Please Login or Register  to view this content.
    Please help, I am dealing with this web page for days without success.
    I add _VIEWSTATE and __EVENTVALIDATION after other parameters now, but I found a horrible thing that after refreshing the web page the value of _VIEWSTATE and __EVENTVALIDATION changed, does this mean that I cannot use the POST method to do the task?

    Currently used code as follow:

    Please Login or Register  to view this content.
    The following is the return

    Please Login or Register  to view this content.

+ Reply to Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 12
    Last Post: 01-14-2014, 01:56 PM
  2. Populate Server Function based on Server Name for Server Inventory
    By mvcp007 in forum Excel Formulas & Functions
    Replies: 0
    Last Post: 05-13-2013, 11:29 AM
  3. [SOLVED] Stat table with formulas for each data element but only when the element exists in table
    By stevemills04 in forum Excel Programming / VBA / Macros
    Replies: 1
    Last Post: 01-28-2013, 05:50 AM
  4. Replies: 1
    Last Post: 04-26-2007, 10:24 PM
  5. Replies: 0
    Last Post: 11-19-2005, 08:10 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Search Engine Friendly URLs by vBSEO 3.6.0 RC 1