Execution time question fopen()

chilliboy

Hi,

I have a script that spiders various URL on a periodic basis, which is set up to spider away on a CRON job.

The script runs through numerous URL's during one exection and filters them on content to a database.

However I am having problems with the execution time of the script as on the occasional fopen(URL, r), the script will hang due to the remote server of the URL being slow. This means the script cannot move onto the next URL while it is still trying to open the current URL.

Is there any way I can put a time limit on the fopen() so that if there is no data passed in a set time the connection is closed and the next URL is tried ?

Any thoughts and ideas on how to solve this - presumably search engines must do something along those lines.

Thanks in advance

theslinky

you can try using sockets instead:

fsockopen() and set a timeout using socket_set_timeout().

you'll need to pass in the GET header:

GET http://www.domain.com/page.html HTTP/1.1\n\n

then read from the socket.

you'll also get the headers with the content, but i'm sure it will be easy to parse them out (or ignore them).

or just try doing socket_set_timeout() and use fopen() and see if that does the trick.

-sridhar

chilliboy

Many thanks for the advise. On having a look at the description of the functions for socket_set_timeout() and fsockopen() they seem to be what I'm looking for. From the descriptions the time limit you put on fsockopen() refers to the the timelimit on the connection opening, and you must use the timelimit in socket_set_timeout() for the timelimit on receiving data via the connection.

However my knowledge of PHP is fairly novice and I don't understand much of the script shown in the manual examples.

I was wondering if anyone could help me write the piece of script I require:

The script I have at the moment is as follows

 	$rf = fopen($Data_URL, "r");
	$Page_Data = fread($rf, $Bytes);
	fclose($rf);

What I would like to do is set a 'timeout' ($Timeout) in seconds - where the data should be from the remote URL within the timeout period else(execute alternative function).

Any help would be much appreciated.

chilliboy

Hi,

Is there anyone who can help with the above. The example it give in the manual is as follows, but due to my lack of php knowledge I'm not sure how to alter the code to what I want.

 <?php
$fp = fsockopen("www.example.com", 80);
if(!$fp) {
    echo "Unable to open\n";
} else {
    fputs($fp,"GET / HTTP/1.0\n\n");
    $start = time();
    socket_set_timeout($fp, 2);
    $res = fread($fp, 2000);
    var_dump(socket_get_status($fp));
    fclose($fp);
    print $res;
}
?>

Any help would be greatly appreciated.

chilliboy

Anyone able to help me with the above - I would be really grateful

Kind Regards