cURL on aspx possible?

kireol

Hi all.

I'm tryng to scrape an internal website at my work that was wrote by someone in .aspx

When I try and scrape the page with cURL, it wont set the values on the form. it just reloads the page. I'm expecting it to set the values that I told it with the POST variables.

I wrote an example on an external site that I found that also uses .aspx. I think this could be any random site wrote in .aspx.

I'm hoping someone has been able to do this before or can figure out why cURL post variables dont work on a .aspx site.

here's a script that loads a page. saves the viewstate variable. then reloads the page and tries setting a form var. but it doesnt. why?

<?php
        $cookie_file_path = "cookies/cookiejar.txt"; // Please set your Cookie File path
        $fp = fopen("$cookie_file_path","w") or die("<BR><B>Unable to open cookie file $mycookiefile for write!<BR>");
        fclose($fp);
        $ch = curl_init();

    $url ="http://www.rebac.net/MembershipDirectorySearch.aspx";
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
    $result = curl_exec ($ch);
    $token = "__VIEWSTATE";
    $result = strstr($result, $token);
    $result = substr($result, strlen($token));
    $endpos = strpos($result, '" />');
    $result = substr($result, 9,$endpos-4);

    $postoptions = "__VIEWSTATE=$result&DropDownListSearchNationalState=MI";
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS,$postfields);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
    $result = curl_exec ($ch);
    echo $result;
    curl_close ($ch);
?>

MarkR

Possible debugging ideas:

Enable error_reporting(E_ALL) and error logging, look at the logs
Enable tracing on the ASPNET server into a log file, and look at that (e.g. using the ASPNET trace functions). You can also set up an application-wide aspnet error handler to trap any exceptions that happen and log them somewhere or email etc.
Run Ethereal and watch the conversation

Then you'll have some idea what's happening.

"It doesn't work" is not sufficiently descriptive.

Mark

kireol

MarkR wrote:
Possible debugging ideas:

Enable error_reporting(E_ALL) and error logging, look at the logs

Enable tracing on the ASPNET server into a log file, and look at that (e.g. using the ASPNET trace functions). You can also set up an application-wide aspnet error handler to trap any exceptions that happen and log them somewhere or email etc.

Run Ethereal and watch the conversation

Then you'll have some idea what's happening.

"It doesn't work" is not sufficiently descriptive.

Mark

Cool, i'll try those ideas. thanks for the tips.

is the ""it doesnt work" is not sufficiently descriptive." part of your signature?

I ask because i clearly stated the problem. "it wont set the values on the form". looks very explicit to me.

MarkR

kireol wrote:
looks very explicit to me.

You didn't say what it DID do , only what it didn't. Sorry if I misinterpreted your question 🙂

Seriously though, you should be checking what it does do rather than what it doesn't. Enabling the above logging will help you to determine that.

A lot of people do post a great deal of stuff (not usually in as much detail as you), then ask "Why doesn't it work?".

This is what I had misread yours as.

Mark