Something wrong with URL grabbing code

andrewm

I created a crawler tha crawls websites and stores certain data from them. It works great for most sites, but when crawling a few sites it errors out, giving me the Max Execution Time error for the lines:

"while ($pos < strlen($result)-20) {"

and

"if (eregi("HREF[ \t\n\rv]=[ \t\n\rv][\"']([^\"'])[\"']*",$tag,$regs)) {"

Can you see anything thats wrong with the following code that could cause problems with some websites?

$urls = array(); 
$links = array(); 
$pos = 0;  

while ($pos < strlen($result)-20)  { 
	$pos =  strpos($result,"<",$pos); 
	$pos++;  

	$lastpos = strpos($result,">",$pos);  

	$tag = substr($result,$pos,$lastpos-$pos); 

if (!strcasecmp(strtok($tag," "),"A") or !strcasecmp(strtok($tag," "),"AREA")) { 
	$pos = $lastpos+1; 
	$linkpos = $pos; 
	$pos = strpos($result,"<",$pos);  

	if (eregi("HREF[ \t\n\rv]*=[ \t\n\rv]*[\"']*([^\"']*)[\"']*",$tag,$regs))  { 

		// Some other checks and then store to $url array if it passes

	} else { 
		$urls[] = "";  
	}  
} 
$pos = $lastpos+1; 
} 

for ($i=0; $i<sizeof($urls); $i++) {

	//store URLS to database
}

andrewm

Also, what does it usually mean when there is a Max Execution time error?

Could it really stall for 1000 seconds (I tried forcing the max execution time with "ini_set("max_execution_time", 1000);") on a pretty simple regular expression?

andrewm

Anyone?

Rohan_Hill

Could it just be that the webpage it's crawling takes too long to load?

andrewm

Could it be that?

I was thinking it would error out with my curl commands if the page couldn't be loaded:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$currenturl);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec ($ch);
curl_close ($ch);

Wouldn't curl error out after the configured amount of time and then $result would just be blank?