using cURL on sites wrote in .aspx possible?

kireol

There's several sites out there that I'm trying to use cURL to scrape. every one that's wrote in .asp wont allow me to scrape. I'm guessing it has something to do with the "VIEWSTATE" variable, but that's just a hunch at this point.

I found a random website that uses .asp and even that one is the same. so I wrote a small script so that maybe someone can help me figure out why I can't get this form to process properly.

If in a browser, you select a state from the drop down for state, it will reload the page like it should. I've tried to redo this with curl, and it doesnt seem to work.

<?php
        $url ="http://www.rebac.net/MembershipDirectorySearch.aspx";
        $postoptions = "DropDownListSearchNationalState=MI";
        $cookie_file_path = "cookies/cookiejar.txt"; // Please set your Cookie File path
        $fp = fopen("$cookie_file_path","w") or die("<BR><B>Unable to open cookie file $mycookiefile for write!<BR>");
        fclose($fp);
        $agent = "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)";
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL,$url);
        curl_setopt($ch, CURLOPT_POST, 1);
        curl_setopt($ch, CURLOPT_POSTFIELDS,$postfields);
        curl_setopt($ch, CURLOPT_USERAGENT, $agent);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
        curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
        curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
        $result = curl_exec ($ch);
        curl_close ($ch);
        echo $result;
?>

kireol

any help?

thorpe

You'll need to be alot clearer with your description of the problem.

MarkR

Please supply:

What you're trying to achieve using the CURL POST
What error message / HTTP response you get from the other site
Please explain why you're trying to forget the user agent header? This is not something you should be doing.

If you're trying to scrape this page without permission, and it stops working, I have no sympathy with you whatsoever.

You're correct in suggesting that ASPX normally require a valid viewstate to correctly process a POST (simulating a button click). To resolve this, simply load the form first and parse the fields out.

Mark

kireol

It's all in my first post. The problem, sample code, how it works manually, and how it doesnt work with code. so I must be reading my own post wrong, it's all there as I see it. so i'll try and expand and maybe I can make it clearer what my problem is.

MarkR wrote:
1. What you're trying to achieve using the CURL POST

I'm trying to make my script run some forms and read the data on my companies internal site that was wrote in .aspx by a 3rd party that doesnt work for my company any more.

MarkR wrote:
2. What error message / HTTP response you get from the other site

As stated in my first post, it's not an error message, but more simply the fact that when my script reloads the form and had tried to change a field, the form comes back unaltered. but do this manually with a browser and it works fine of course.

If I manually use a browser and change a form variable.....the page reloads and a form variable is set.

MarkR wrote:
3. Please explain why you're trying to forget the user agent header? This is not something you should be doing.

I'm not sure what yer getting at here. If you are talking about

$agent = "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)";
curl_setopt($ch, CURLOPT_USERAGENT, $agent);

that code, I'm trying to set it to this value as I was told once you should do this so it looks like a real browser.

MarkR wrote:
If you're trying to scrape this page without permission, and it stops working, I have no sympathy with you whatsoever.

I dont have permission as I alluded to in my first post. But since the sites that I'm really working on are on my works intranet, and are private, I found the first site that I could find that would work as an example here. Better to work on a real example instead of abstract ideas IMHO

MarkR wrote:
You're correct in suggesting that ASPX normally require a valid viewstate to correctly process a POST (simulating a button click). To resolve this, simply load the form first and parse the fields out.

There's probably my problem. When I look at the viewstate values in my firefox browser source, it's blank. and I was stumped at that point, which is why I came here. I was having trouble getting the value of viewstate to put it in my POSTFIELDS section. I'm wiling to take a stab once I get this set, it'll work. but having issues here just to get it.

thanks,

Chris

kireol

OK,

so I got the viewstate part saved into a new variable. and then I set that as a post variable. When I try loading the page with the state field changed, it's not loading up as changed.

Here's my new code, which i think is closer, as it loads the default page, saves the viewstate and sends the viewstate and state fields to the page, however, Michigan is not set when it reloads. why is that? that's my "error"

<?php
        $cookie_file_path = "cookies/cookiejar.txt"; // Please set your Cookie File path
        $fp = fopen("$cookie_file_path","w") or die("<BR><B>Unable to open cookie file $mycookiefile for write!<BR>");
        fclose($fp);
        $ch = curl_init();

    $url ="http://www.rebac.net/MembershipDirectorySearch.aspx";
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
    $result = curl_exec ($ch);
    $token = "__VIEWSTATE";
    $result = strstr($result, $token);
    $result = substr($result, strlen($token));
    $endpos = strpos($result, '" />');
    $result = substr($result, 9,$endpos-4);

    $postoptions = "__VIEWSTATE=$result&DropDownListSearchNationalState=MI";
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS,$postfields);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
    $result = curl_exec ($ch);
    echo $result;
    curl_close ($ch);
?>