I need to create a script that can POST to a remote form.
From the standpoint of my ability, I'd probably prefer to do this in PHP. Would cURL be the logical approach?
I've done some searching and some people are recommending Python and its 'mechanize' library for this sort of thing. I know a little bit about Python ... would the difficult of doing this in PHP w/cURL justify the time it would take to get my Python-{spam} up to snuff?
dalecosp;11020527 wrote:Would cURL be the logical approach?
That'd probably be my first choice, only because using cURL for such a task is actually pretty easy - you might just have to take a few minutes to read over the cURL man pages (heavily concentrating on [man]curl_setopt/man more than likely) to understand what/how to set up your cURL "session."
Depending upon the complexity of this POST operation, you could even use [man]file_get_contents/man to do it (although I think the cURL approach would be more straightforward and/or easier to understand).
I've done this sort of thing with cURL in php, and tbh I don't feel its that hard. Knowing you, a day of research and you're good to go. Take a look at [man]curl_setopt[/man] (notably: CURLOPT_POST and CURLOPT_POSTFIELDS) as that is where you set everything up. You call init, set options, and call exec. If you need to get the data returned you assign the return value of [man]curl_exec[/man] to a variable. You should use [man]curl_getinfo[/man] to check the status code of the request.
Don't know if it helps or not, but here's the primary function of one of my classes that interacts with a 3rd party api.
private function callApi ($url, $params = array(), $method = 'get')
{
$data = null;
$headers = array();
$headers[] = "Content-type: application/json";
$headers[] = "api_key: " . $this->API_KEY;
if (isset($this->AUTH_KEY) && ! empty($this->AUTH_KEY)) {
$headers[] = "auth_token: " . $this->AUTH_KEY;
}
$url = (self::BASE_URI . $url);
$curl = curl_init();
curl_setopt($curl, CURLOPT_TIMEOUT, 5); // 5 second timeout
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true); // return the result
// on success, rather
// than just TRUE
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
if ($method == 'get' && ! empty($params)) {
// set query params if method is get
$url = ($url . '?' . http_build_query($params));
}
elseif ($method == 'post') {
// set post data if the method is post
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, json_encode($params));
}
elseif ($method == 'delete') {
// set post data if the method is delete
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "DELETE");
curl_setopt($curl, CURLOPT_POSTFIELDS, json_encode($params));
}
curl_setopt($curl, CURLOPT_URL, $url);
// make the request
$response = curl_exec($curl);
$response_info = curl_getinfo($curl);
// handle the response based on the http code
if ($response_info['http_code'] == 0) {
throw new Exception("TIMEOUT: api call to " . $url .
" took more than 5s to return");
}
elseif ($response_info['http_code'] == 200) {
$data = json_decode($response);
}
elseif ($response_info['http_code'] == 401) {
throw new Exception("Unauthorized API request to " . $url . ": " .
json_decode($response)->message);
}
elseif ($response_info['http_code'] == 404) {
$data = null;
}
else {
throw new Exception("Can't connect to the api: " . $url .
" response code: " . $response_info['http_code']);
}
return $data;
}
Thanks, gents! 
Derokorian, it's rather helpful indeed! About this line:
curl_setopt($curl, CURLOPT_POSTFIELDS, json_encode($params));Any particular reason this is being encoded in JSON? Is that necessary for all POSTed data?
dalecosp;11020635 wrote:Any particular reason this is being encoded in JSON? Is that necessary for all POSTed data?
Derkorian mentioned that he was working with an API, so I'm guessing that this API expected JSON-encoded data in the body of the requests. You wouldn't want to do this if, for example, you were attempting to simulate a normal POST request from a browser to a webserver.
bradgrafelman;11020655 wrote:Derkorian mentioned that he was working with an API, so I'm guessing that this API expected JSON-encoded data in the body of the requests. You wouldn't want to do this if, for example, you were attempting to simulate a normal POST request from a browser to a webserver.
+1 This is exactly the reason, the docs for the API I'm using specifically says it expects json encoded string. Not exactly sure why, but that's what they want. 
If I were to hazard a guess on that, (and my guesses are often hazardous), it'd have something to do with their service being tuned for AJAX from someplace and they wanted to reuse the code.
At any rate:
bradgrafelman wrote:normal POST request
Given that the target pages are ASP with all that JS __doPostBack() crap, I'm not sure anything about it's what I'd call "normal", but I am making some progress. Slowly.
My boss is OOO today and left me in charge of bugging the Long-Distance company whose link to us has been down one day short of a week at this point. Fun, fun, fun ....
Thanks, gentlemen!
Looks like I've made it work!
Hey dale, any chance you'd show me how you emulated the .net postback? I've been trying to adapt a tool I made to work with our company site, however I don't know how to "follow the links" if you know what I mean.
It's nasty and dirty. You've got to post back all those hidden fields...whatever they may be. In this application they are the same for every page: EVENTVALIDATION, VIEWSTATE, and of course the EVENTARGUMENT and EVENTTARGET fields:
//$d is the HTML source I've read from Page$1
$regexp1 = '%name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="[/a-zA-Z0-9+=]{1,1000}"%';
$regexp2 = '%name="__VIEWSTATE" id="__VIEWSTATE" value="[/a-zA-Z0-9+=]{1,50000}"%';
preg_match($regexp1, $d, $eventvalidation);
preg_match($regexp2, $d, $viewstate);
$out = explode("value=",$eventvalidation[0]);
$evt_validation = trim($out[1],'"'); // like my nasty hacks? I should really get better with preg_match, eh?
$out = explode("value=",$viewstate[0]);
$view_state = trim($out[1],'"');
$evt_target = 'ctl00$ContentPlaceHolder1$gv_results';
$evt_argument = 'Page$'.++$curr_page;So, for subsequent pages:
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
//ASP servers think they know everything; if you don't spoof a JS-enabled client they don't send all the hidden fields
curl_setopt($ch, CURLOPT_USERAGENT, ('Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0'));
// set up the CURL POST STUFF HERE
curl_setopt($ch, CURLOPT_POST,1);
$poster_ray = array("__EVENTTARGET"=>"$evt_target","__EVENTARGUMENT"=>"$evt_argument",
"__EVENTVALIDATION"=>$evt_validation,"__VIEWSTATE"=>$view_state);
curl_setopt($ch, CURLOPT_POSTFIELDS,$poster_ray);HTH,
dalecosp;11020877 wrote:// like my nasty hacks? I should really get better with preg_match, eh?
An even better method might be to get better with [man]DOM[/man] so that you could locate the <form>'s children (hidden or otherwise) and directly add them into your cURL array of fields to POST. From there, you could then overwrite (or add?) any values you desire to specify; the remaining fields (such as the hidden inputs) would then get POST'ed with whatever default values they contained in the original page with the form. Granted, this extra flexibility might be rather pointless depending upon the nature of the remote form.
Either way... for your PCRE stuff, why are you doing that exploding/trimming rather than simply placing a pair of ()'s around the part of the pattern you want to extract? (In other words, why aren't you grouping the data you want so that the regexp engine will give you exactly that?)