Help with scraping an ASPX LOGIN required site

[dbeck83]

Hi, I need to scrape information from a website that requires ASPX login (the first page is a form which asks for user name and password). I am just wondering which method I should use to do this. Can I use cURL to fill in the form and send the login information to the server then access the page or can I login first to generate a cookie and then use that cookie to access the page (behind the login) I need to scrape from. The site I am trying to scrape is www.knowledge.reuters.com. Any help would be greatly appreciated! Thank you!!

[bradgrafelman]

Eh... is this even allowed by them? 😉

[robert_w_ark]

Well, in this case you would have to set the user agent to MSIE (Or something like "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)") -- You can do this by using PHP's function header();. Also, you will need to use curl, so make sure the server that you use the script with has the curl extension enabled, If not, a simple request could have it enabled.