[RESOLVED] Search tab delimited file

Abarak

I am having one heck of a time searching through a LONG tab delimited text file. As for right now, I am comparing the input value of the function while going through EVERY line.

function tabList($Find) {
	$readfile = file("inputFile.php");

// Create a loop that will read all elements of the array and print out
// each field of the tab-delimited text file		
for ($k=0; $k<=count($readfile)-1; $k++) {
	$fields = split("\t",$readfile[$k]);
	if($fields[14] == $Find) {
		$found = true;
		$price = $fields[10];
	}
}

if(!$found)
	$price = "Not Found";

$rtnArray = array("Name" => "-Name Here-", "Price" => $price);
return $rtnArray;
}

Man, going through 500,000+ lines is killing the above and forcing it to timeout. If ANYONE can suggest a faster way of searching through the tab list I would be most appreciated. So far, this is really making my monday suck.

And no, it is a tab delimited list and there is nothing I can do about it.

leatherback

Hey Chad,

I have NO CLUE whether this works in your situation, and whether it is faster..

Have you tried:

-> Subsetting the file, so retrieve the thing, and get the 14th colum out, and through the rest of the file from memory?
-> Is there a system in the file? Could you do a splitting by row, so test the 100.000 row, if the value is less, you search before, else after? Maybe even by sorting the thing? (Although I am afrain that sorting is inefficient)
-> Have you tried reading the file in a spcific format? [man]fscanf[/man]

Installer

This isn't a complete solution, but if you do this you'll at least occasionally save some time:

if ($fields[14] == $Find) { 
    $found = true; 
    $price = $fields[10];
    break; 
}

What you really need is a database solution.

Abarak

Thanks for the information and sorry about the caps. Some times I just get a little carried away... 😃

The tab delimited list is pulled from an external server every 3rd day using a cron job. At this point I believe the best option would be to toss the entire file in a MySQL DB every time the file is pulled.

Thanks for the assistance!

Weedpacket

The database approach would help (it would ease searching somewhat, assuming you can get it into the database in the first place without too much grief - depends on which DBMS you pick), but I can suggest a couple of other interim measures that might help (as well as Installer's suggestion of break, which is just a plain good idea):

Use [man]fgetcsv[/man] to read the file; setting the delimiter to a tab. This would entail using a while loop instead of a for; because you'll be processing the file as you read it, instead of reading it in full first (which could also impact on time as well, if fitting such a big lump of data into memory ends up needing a lot of shuffling around to make room)
Adding a [man]set_time_limit[/man] inside the loop - just a short one that guarantees enough time for one more iteration.

Installer

I wrote a script to test different ways to do this and was surprised at some of the results. Both the "fgets()" and "file()" methods (along with "stream_get_line()" on v.5), compared to "fgetcsv()", came out way faster on Windows, running in 1/5 to 1/4 the time. I expected the other way around. On Linux it was pretty much a wash, all coming within 10% of each other. That surprised me a little, too. This was with a local text file; the results with a remote file, although they varied quite a bit, were generally the same. BTW, using "explode()" was a lot faster than using "split()" (taking 1/3 the time).

If anyone is interested and wants to check the code for me (I'm not an expert speed-tester) or try it themselves, just say so and I'll post it.

Given all that, I think you'll find a great improvement doing it like this:

while (!feof($fp)) {
    $fields = fgetcsv($fp, 512, "\t");
    if ($fields[14] == $find) {
        $found = true;
        $price = fields[10];
        break;
    }
}

Or, on Windows:

while (!feof($fp)) {
    $line = fgets($fp, 512);
    $fields = explode("\t", $line); 
    if ($fields[14] == $find) {
        $found = true;
        $price = fields[10];
        break;
    }
}