Will PHP be able to parse 700000+ line of logs ?

ftrudeau

Yup, that's what I plan to do ...
Those are radius logs, are approx. 23.6 Mb uncompressed, and 2.1 Mb compressed, with gzip.

I was wondering if someone had any idea how to actually make PHP parse this huge log file, without using all my computer's memory =)

Any pointers, please ?

ednark

you can set these two variables (amung others) with [man]ini_set/man

memory_limit
max_execution_time

so you can set a small memory limit and a long execution time for your script

dopey

Depends on how you write it as well...

i.e. don't do

// get contents of a file into a string
$filename = "/usr/local/something.txt";
$fd = fopen ($filename, "r");
$contents = fread ($fd, filesize ($filename));
fclose ($fd);

🙂

something along the lines of

$filename = "/usr/local/something.txt";
$fd = fopen ($filename, "r");

while( $contents = fread ($fd, 1024) )
{
    //do wonderful things to $contents and prob. put somewhere
}
fclose ($fd);

should be ok...

Cheers,
Andy