socket_set_timeout help needed

chilliboy

Hi,

I was wondering if there was anyone over here that could help. I posted the below and got a part answer, but it was to complex for my more basic understanding. If anyone can help with the below I would be eternally grateful. I have copied the questions and part answer , so that you can follow it through. I'm really looking for someone to simplify what the guy answered to the initial question and guide me through what I should do - code I should use. As B4 I would be really grateful for any help on this one.

The Initial Question

Hi,

I have a script that spiders various URL on a periodic basis, which is set up to spider away on a CRON job.

The script runs through numerous URL's during one exection and filters them on content to a database.

However I am having problems with the execution time of the script as on the occasional fopen(URL, r), the script will hang due to the remote server of the URL being slow. This means the script cannot move onto the next URL while it is still trying to open the current URL.

Is there any way I can put a time limit on the fopen() so that if there is no data passed in a set time the connection is closed and the next URL is tried ?

Any thoughts and ideas on how to solve this - presumably search engines must do something along those lines.

Thanks in advance

The REPLY

you can try using sockets instead:

fsockopen() and set a timeout using socket_set_timeout().

you'll need to pass in the GET header:

GET http://www.domain.com/page.html HTTP/1.1\n\n

then read from the socket.

you'll also get the headers with the content, but i'm sure it will be easy to parse them out (or ignore them).

or just try doing socket_set_timeout() and use fopen() and see if that does the trick.

-sridhar

The Follow Up Question

Many thanks for the advise. On having a look at the description of the functions for socket_set_timeout() and fsockopen() they seem to be what I'm looking for. From the descriptions the time limit you put on fsockopen() refers to the the timelimit on the connection opening, and you must use the timelimit in socket_set_timeout() for the timelimit on receiving data via the connection.

However my knowledge of PHP is fairly novice and I don't understand much of the script shown in the manual examples.

I was wondering if anyone could help me write the piece of script I require:

The script I have at the moment is as follows

PHP:--------------------------------------------------------------------------------
$rf = fopen($Data_URL, "r");
$Page_Data = fread($rf, $Bytes);
fclose($rf);

What I would like to do is set a 'timeout' ($Timeout) in seconds - where the data should be from the remote URL within the timeout period else(execute alternative function).

Any help would be much appreciated.

ozzythaman

after the GET line, use the "host [domain]" ... many servers depend on these because many servers are serving several hosts.. as in 5 domains -> 1 ip (one server)

and by the way, the timout setting with sockets do not work most of the time... this is a bug that has been reported like 100 times to php.net and has been said to be resolved 100 times too..

but it still doesnt work...

chilliboy

Hi,

This is the final script I have come up with

function Read_Page($Source_Page_Code, $Base_Source_URL)
{	
global $Bytes, $Soc_Open_Secs, $Soc_Read_Secs;

$Data_URL = $Base_Source_URL[Start] . $Source_Page_Code;
echo 'Data URL  =' .  $Data_URL . "<br>\n";
$fp = fsockopen($Data_URL, $Soc_Open_Secs); //open connection
if(!$fp)
{
    Error_Message("Socket did not open in $Soc_Open_Secs Seconds");
}
else 
{
    fputs($fp, "GET / HTTP/1.0\n\n"); //send HTTP request
    socket_set_timeout($fp, $Soc_Read_Secs);
    while (!feof($fp)) 
	{
        $Page_Data .= fread($fp, $Bytes[Team_Page]); //read page contents
    }
    fclose($fp);
}
if($Page_Data)
{ 
	//echo "This is Page_Data = $Page_Data \n<br>"; #############****************************************
	return $Page_Data;
}
}

However, when I set $Soc_Read_Secs to x number of seconds (eg 10 secs), it throws up the error almost instantly i.e from my understanding it is not waiting for the 10 seconds to elapse. Is this a problem with my script or is it a bug in PHP ?

If it is a bug, then does anyone know a simple work around where I can make a script to do what I want : Try to read data from a page via http in 'x' seconds - if no data has been read in that time the script simply moves onto the next URL to read (rather than hanging until it timeouts or eventually reads the data after a very long time).

Many thanks for the help.