Dealing with multi-MB files

Anon

I have a log file that can easily grow to 10+ megs. PHP seriously chokes on it - just reading each line causes the script to timeout at 30 seconds. I thought PHP would be able to speed through a file like this - after all, analog can rip through a 500 MB log file in about 5 seconds.

Here's two variations of my code:

$fp = fopen($stats, 'r');

while (fscanf($fp, "%s\t%s\n", $date, $id) == 2) {
$stats[$id][$date]++;
...
}

fclose($fp);

///////////////////////////////
$file = file($stats);

foreach($file as $line) {
list($date, $id) = explode("\t", $line);
...
}

[deleted]

"PHP seriously chokes on it "

What you need to realize is that PHP is a web-scripting language. It is not designed to parse logsfiles.

Use PERL for that, or an executable.
Or even better, write your log entires directly into a database.

PS Analog? 500Mb/5seconds? 100mb/second? Does your harddisk even read that fast? :-)

Anon

I'm trying to write a database-less version 🙂. Appending to the log file is fast enough, but generating reports from the log file is proving to be too much for PHP. Perhaps I'll try Python, PERL, or C++ - not sure if they'll deal with huge files much better.

"PS Analog? 500Mb/5seconds? 100mb/second? Does your harddisk even read that fast? :-)"

Honest 🙂. It only takes about 5 seconds to chew through 500-600 MB apache logs. I don't know how (yet)...

[deleted]

"I'm trying to write a database-less version 🙂."

lol and the fact that manu have come before you and many have failed doesn't put you off?

I guess there will always be flat-file-freaks :-)

"I don't know how (yet)..."

Probably because it doesn't.
it probably scans the file and the entries and only processes the new ones.

Anon

"I'm trying to write a database-less version 🙂."

lol and the fact that manu have come before you and many have failed doesn't put you off?

I guess there will always be flat-file-freaks :-)

Not me, but there are script users who don't have a clue/don't want to use one of them complicated databases contraptions :-P

"I don't know how (yet)..."

Probably because it doesn't.
it probably scans the file and the entries and only processes the new ones.

Don't think so - even if it did just scan it, it still reads/seeks every line. Same thing in PHP chokes it. You've seen apache logs - they're not sorted at all. You'd have to read every line.

Anon

it could just be your approach at processing the file that is wrong. having a 10+ meg array could certainly slow things down. im sure you can come up with a way of processing the file that doesnt require the whole thing to be in an array at once.

im sure analog doesnt try and load your entire 500 MB log file into memory...

paul
http://xhawk.net

Anon

Nope - with fscanf and fgets only a line at a time is read into memory. You can use/ignore those lines as you wish. The problem is seeks through the file, not memory.