fopen() slow on first run

[tkm256]

I have a script that searches about 1500 files for keywords (http://indiana.edu/~liblilly/lilly/mss/test/test.php). The first time a search is run, it can take upwards of 20 seconds to return results. But after that, it takes only 5 for the exact same process.

After a lot of tests, I've narrowed the problem down to one line: fopen(). For example, this is how long it takes five of my files to open on the first run of a search for 'Picasso':

0.002117 s
0.005830 s
0.003877 s
0.004670 s
0.007206 s

And immediately after:

0.000501 s
0.000516 s
0.000570 s
0.000646 s
0.000511 s

There's an entire zero missing in the first run, which for every 100 files adds an extra second => 15 second lag.

Has anyone had this issue with fopen() before? As far as I know, the server isn't caching the opened files anywhere. The rest of my search options are speedy, but this is a significant enough problem to keep people from using my search at all.

[dagon]

if i was doing this a would probably go for grep over the php way, can you post a small example of the code you are using, something i can test locally.

[tkm256]

Here's the full-text search part of the code. First I have the function, then the call to it searching every file in the array $all_files. The important part is right up top on line 3.

function file_find($file, $phpfilename, $num_ands, $keywords_core, $keywords_boolean, $phpfilenames, $ascii_limit) {
	$filetype = substr($file, -3, 3);
	$fp = fopen($file, "r") or die ("unable to read");
		$invpagecount = -1;
		$stop = false;
		$anynot = false;
		$anyor = false;
		$num_ands_found = 0;
		$remaining = $keywords_core;
		$remaining_relations = $keywords_boolean;
		$remaining_ascii = $ascii_limit;
		$ftsearch_results = array();		

	while (!$stop) {
	//get line of text
		$line = strtolower(fgets($fp));
		if ($line == false) {$stop = true;}
		else {
			//if inventory file, keep track of which page
				if ($filetype == 'xml') {
					if (substr($line, 0, 5) == '<page') {
						$invpagecount++;
					}
				}
			//find keywords in line, add filename to array	
			foreach ($remaining as $keyword) {

				$indexOf = array_search($keyword, $remaining);					
				$operator = $remaining_relations[$indexOf];
				$ascii_ceiling = $remaining_ascii[$indexOf];
				$beginword = strpos($line,$keyword);
				$wordlength = strlen($keyword);

				//determine ASCII limits
				$prelim = 65;
				$postlim = 65;
				//if asterisk in front, allow all characters
				if ($ascii_ceiling == 1) {
					$prelim = 255;
				}
				//if asterisk on end, allow all characters
				else if ($ascii_ceiling == 2) {
					$postlim = 255;
				}

				//if "not" word in line, stop loop and flag "not"	
					if ($operator == '1') {
						if ($beginword) {
							$pre = ord(substr($line, $beginword-1, 1));
							$post = ord(substr($line, $beginword+$wordlength, 1));
							if ($pre < $prelim && $post < $postlim) {
								$anynot = true;
								$stop = true;
							}
						}
					}

				//if "and" word in line, remove from remaining terms and add to counter
				//this is necessary because ALL and words need to be present in file, must search for others & counter must match total number
					else if ($operator == '2') {
						if($beginword) {
						$pre = ord(substr($line, $beginword-1, 1));
						$post = ord(substr($line, $beginword+$wordlength, 1));
						if ($pre < $prelim && $post < $postlim) {
							$num_ands_found++;
							$invmarker = $invpagecount;
							unset($remaining[$indexOf]);
							array_values($remaining);
							unset($remaining_relations[$indexOf]);
							array_values($remaining_relations);
							unset($remaining_ascii[$indexOf]);
							array_values($remaining_ascii);
							}
						}
					}

				//if "or" word in line, flag "or"
					else if ($operator == '3') {
						if ($beginword) {
						$pre = ord(substr($line, $beginword-1, 1));
						$post = ord(substr($line, $beginword+$wordlength, 1));
							if ($pre < $prelim && $post < $postlim) {
								$anyor = true;
								$invmarker = $invpagecount;
							}
						}
					}
				}
			}
		}
	fclose($fp);
	if ($anynot == false && $anyor == true && $num_ands_found == $num_ands) {
		$ftsearch_results[$phpfilename] = $invmarker;
	}
	array_unique($ftsearch_results);
	return $ftsearch_results;
}

$allftsearch_results = array();
//Run file_find on descriptions and add file names to allftsearch_results
for ($i = 0; $i <= count($all_files); $i++) {
	$description_search_result = file_find('mss/'.$all_files[$i], $phpfilenames[$i], $num_ands, $keywords_core, $keywords_boolean, $phpfilenames, $ascii_limit);
	foreach ($description_search_result as $keyfilename => $invpagevalue) array_push($allftsearch_results, $keyfilename);
	}