Using PHP to scrape a site, emulating netscape or IE

darren_wheatley

Hi there 🙂

I have been using php for a little while and it goes ok!!!

I am trying to write a piece of code that will allow me to gather information off another site (not phpbuilder.com 😉. Using PHP and the file command it's pretty easy:

$strURL = "http://www.phpbuilder.com/";
$arrDataHTML = file($strURL);

Doing this on my own server, I notice that when I use file() to get information the HTTP_USER_AGENT that is logged includes the name "PHP".

Is there a way to mask this, or emulate a netscape or IE user agent so that it doesn't look as though it's PHP scraping, rather just another web browser?

Thanks in advance,

Darren

PyroX

use sockets, send the actual request headers that ie does.

suntra

This is how you could make it :

<?
$contenu = array(); // variavle that will contain the contents of the site
$svr = "server.com"; // server address, WITHOUT the http://
$port = 80; // port
$errno = "";
$errstr = "";
$fch = "/directory/file.extension"; // the file you want to get, WITHOUT the server
$userAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)";
$enr = false;
$fp = fsockopen($svr, $port, $errno, $errstr, 30);
if(!$fp) {
	echo "$errstr ($errno)<br>\n";
} else {
	fputs($fp, "GET $fch HTTP/1.0\r\nHost: $svr\r\nUser-agent: $userAgent\r\n\r\n");
	while(!feof($fp)) {
		$lgn = fgets($fp, 4096);
		if($lgn == "\r\n" && !$enr) {
			$enr = true;
		}
		if($enr && $lgn !="\r\n") {
			$contenu[] = $lgn;
		}
		fclose($fp);
	}
}
?>

darren_wheatley

suntra,

Thanks for your reply.

This looks pretty cool... thanks for the detailed example, you have saved me heaps of time.

I'll give it a go!

Pyrox... just keeping the stats up?

Thanks a bunch,

Darren

tomhath

Pyrox... just keeping the stats up?

No. You asked a question and he answered it:

Question: Is there a way to mask this, or emulate a netscape or IE user agent so that it doesn't look as though it's PHP scraping, rather just another web browser?

Answer: use sockets, send the actual request headers that ie does.

PyroX

suntra

You seem to know http response headers, here is a ? for ya,

Lets say the server tries to set a cookie:

Set-Cookie: BrowserTest=Success%3f; domain=.testurl.com; path=/

How would I format the response to tell it, yes I have your @&#% cookie! ?

PyroX

Nevermin, the all powerful google answers all:
http://wp.netscape.com/newsref/std/cookie_spec.html

Weedpacket

I just found these two lines in my copy of php.ini;

; Define the user agent for php to send
;user_agent="PHP"

I just wondered if they had anything to do with anything...

chrisjj

Thanks suntra, but your code has a fault - the fclose should be outside rather than inside the while{}, i.e.

$contenu = array(); // variavle that will contain the contents of the site
$svr = "server.com"; // server address, WITHOUT the [url]http://[/url]
$port = 80; // port
$errno = "";
$errstr = "";
$fch = "/directory/file.extension"; // the file you want to get, WITHOUT the server
$userAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)";
$enr = false;
$fp = fsockopen($svr, $port, $errno, $errstr, 30);
if(!$fp) {
echo "$errstr ($errno)<br>\n";
} else {
fputs($fp, "GET $fch HTTP/1.0\r\nHost: $svr\r\nUser-agent: $userAgent\r\n\r\n");
while(!feof($fp)) {
$lgn = fgets($fp, 4096);
if($lgn == "\r\n" && !$enr) {
$enr = true;
}
if($enr && $lgn !="\r\n") {
$contenu[] = $lgn;
}
}
fclose($fp);
}