Curl with .aspx file login

[cjm]

Hello,

I am trying to use CURL to scrape the data from a page that requires a user to log in.

If I am logged in the browser I am using seems to store information somewhere that I am logged in. If I use any of the buttons on this aspx page via the browser to perform a search for the data I need, the link does not change but the html displayed does.

So is there any way to get the html that the browser is displaying based on choices made by the user for a .aspx file? If I just use CURL to go to that link it will always take me to the login screen as if the user has not done anything yet.

If there is another tool that would be better for scraping this type of data please let me know.

[cjm]

To simplify it all, I am having trouble getting CURL to work with the saved states of .aspx files through session variables or however the aspx files do it. Can anyone help me?

[WanderingZombie]

does anyone have a solution? the ASP server appears to ignore posted values until I also send some VIEWSTATE and EVENTVALIDATION values along too... and then the server gives me a 500 error :mad: Frustrated am I!

[shawty]

In a nutshell, unless the site is using http security then you can forget it.

The problem arises from the fact that the .NET runtime actually handles the login/authentication of the aspx application, and it manipulates the cookie and it's values.

Also, the ONLY bit of data thats in the cookie for an ASP.NET app is an MD5 encrypted session ID, if you try to change the ID, it will then not match the re-encrypted versions back on the server (Which is re-encrypted every request) and .NET will then destroy the session and log the user out.

IF you have access to the actual web application, then it's a trivial task to make some small changes to the web.config file, and temporarily turn off the auth system, then do your scrape with curl and php, then undo the changes and re-enable the .NET auth system, failing that the only other way is to log-in save a local html only copy of the page your interested in, then run your php script on that to convert the data.

if the site is secured with a normal http password (The type that makes the browser pop up a password dialog) then you simply just specify the username and password in your URL string when you pass it to curl. EG:

http://user:[e-mail]/

Cheers

Shawty