Reading a 300MB file takes up lots of memory. Is there a better way than FILE ?????

phproy

Hi guys,

If I need to get Line 120 to Line 140 of a hypothetical 300MB text file,

I can do a
$lines = file('textfile.txt');
That will effectively load the whole file and parse them as lines array for me to extract line 120 -140 easily.
Now that is of course a really inefficient way. If you are unlucky, you might get banned by your webhost for being a nuisance.

Is there a way to move the file pointer to line 120 and read 20 lines from there instead of the inefficient way? Can we use FSEEK?

Let's start the discussion!

leatherback

sorry.. see this thread

phproy

Hi Leatherback.

Thank you for pointing me to that thread.

phproy

What a pity the whole thread was useless.
A bunch of posts with no real solution to efficiently access a flat file.

I am looking at dealing with an apache log actually.

drew010

i just put this together since ive seen questions like this before. Im not sure how efficient and fast this will run on a 300MB file, I just put 15 lines in a file each like "Line x" where x is the line number.

<?php

$fp = fopen("lines.txt", "r");
lineseek($fp, 10);
echo fgets($fp);

function lineseek($filepointer, $line)
{
  fseek($filepointer, 0);
  $curline = 1;

  do {
    fgets($filepointer);
    ++$curline;
  } while ($curline < $line);
}

?>

in my test, it outputs Line 10, demonstrating it moved the file pointer to the beginning of line 10, as specified in the function.

if you wouldnt mind, post back telling if it runs fairly quickly or not. thanks

Drakla

Drew - that method is really par for the course, a classic of parallel invention. I use a similar technique for dealing with chunked mode server output that deals with one character at a time and it's not too sloppy. Your file system will probably buffer chunks anyway so you'll be searching memory and it'll whizz along.

phproy

Hi Drew,

Your method is for seeking from line one.

If I wanted to extract line 120-140 I will still have to load from line 1 with your method.
That is still not really efficient.

Drakla

Drew's method will save on memory, which is what you asked for, but will still take time to seek, the only way you can have more efficient seeking is with fixed length rows or a separate index file with the offsets of each row start. If that were the case you could instantly fseek to the right row.

Weedpacket

Drakla's pretty much got the situation summed up. If the lines of the file can be different lengths, then the only way to find line 120 of the file involves locating the first 120 newline characters in the file, which means reading all of the characters in the file, counting newlines as you find them until you get to 120.

drew010

yeah actually at first when i wrote it, i used fgetc reading one character at a time checking to see if it was a newline (\n) and if so increment the line variable, thats why i thought it may run slow, then i remembered fgets would read 1024 bytes or an entire line, which ever came first, so i edited the post. pretty much this should run faster than using file since it doesnt store the contents in any variable, and it just moves the file pointer to the desired line, that way once the pointer is at line 120, you can continue read as many or as little lines as you want from there.

so what you want to do is...

//...
lineseek($fp, 120);

$lines = array();

for ($i = 0; $i < 20; ++$i) {
  $lines[] = trim(fgets($fp));  //read one line, trim \n, push it to lines array
}

print_r($lines);  //should print out lines 120-140

edit:::
tested it on a 60mb video file with 147558 lines. seeking to line 56500 always took 0.11 seconds, with variations in the decimal beyond that point, and seeking to ilne 100,000 took an average of 0.19 seconds. I didnt check the memory usage, but it shouldn't be much different than execution of a "normal" php script.