How to parse remote webpage, loggin in?

tibbs14

I want to parse a remote webpage not on my server. The problem is the page I want to get too needs to have you loged in before you can view it. It uses cookies so once you are logged in you can go back. So I can't use just an fopen because it just receives the login page. It uses a phtml page and a post html method to post the username and password. Any help or a push in the right direction would be appreciated. Thanks.

trooper

Is this not a good opporunity for Curl

Curl

khurramijaz

If you are using PHP 4.3 with stream support you can post data with fopen . ( See stream functions and stream contexts) . Also you can get the HTTP headers sent in the reply from the server ( Set-Cookie is the one you are interested in ) . Use these cookies to post them back to the server (not http post but as a Cookie: header).

If the server is php, java etc powered you can use the session string in the Querystring for the variable. e.g. profile.php?PHPSESSID=.... The session id you got from the server.

CURL is the best option. Very quick and optimised. I have seen it 10 times faster than using sockets urself. (Note: PHP 4.3 has excelent stream support , may you can optimize it to that level, but not worth that effort).

Another alternative is to use Snoopy class just made for this purpose(easiest one). You just need to learn 1-2 methods.

tibbs14

Ok I am using curl now to send a cookie to the webserver. However I do not know what values to send. I have the cookie file IE generates, but it is not in the same format as you would send to curl. So how do I translate that cookie file so I can send it to the website? Thanks.

smorosanu

use curl. You can make post, etc

function curl_string ($url,$user_agent){

$ch = curl_init(); 
curl_setopt ($ch, CURLOPT_URL, $url); 
curl_setopt ($ch, CURLOPT_USERAGENT, $user_agent); 
curl_setopt ($ch, CURLOPT_HTTPHEADER, 1); 
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, "C:\cookie.txt");
curl_setopt ($ch, CURLOPT_COOKIEFILE, "C:\cookie.txt");   
curl_setopt ($ch, CURLOPT_TIMEOUT, 120);

$result = curl_exec ($ch); 
            curl_close($ch);
return $result;

}

this get a page into a string using cookies pretending to be a user agent
You can make post etc, see
http://www.php.net/manual/en/function.curl-setopt.php

To get a page from trow a proxy, I have submited a function to php.net
see this page:
http://www.php.net/manual/en/ref.curl.php (sorin)