Working with extremly large files.

[shawjames]

I'm writing a php script to analyze my server log files and I keep running into errors. I'm guessing this is because the file is a HUGE text file. (over 65meg)

I am trying to read it in like this:

while(!eof($fp))
	$buffer[] = chop(fgets($fp, 4096));

I dont want to break the log file down any further...

Anyone have an idea?

[Weedpacket]

Without more information there are any number of possibilities: Memory or timeout limits set in php.ini are the first that spring to mind. (Obviously not a parse error; eof() is obviously a typo.)

What sort of errors?

Is this a local file, a remote file, opened through fopen() or fsockopen()...?

Does it work with small files? At which point do they get "too large"?

[shawjames]

This is my script, it reads the file apr_2003.log into the array $buffer.

$fp = fopen("apr_2003.log", "r");
$count = 0;
//while($count < 1000) {
while (!feof($fp)){
$buffer[] = chop(fgets($fp, 4096));
$count = $count +1;
}

If I use the while with the $count<1000, it seems to work just fine... but only reads the first 1000 lines. If I use the !feof, it loads for awhile then just comes up with a blank page. No error messages, which happens when there is an error on my webserver.

Any ideas?

[Weedpacket]

What you've posted isn't the whole of the code, obviously...

In different situations I get different errors; PHP times out, the server times out, the browser times out. If I turn off error reporting, the first fails silently after (by default) thirty seconds.

[badboy]

I need to read an extremely large log file (349 M😎 and remove old entry (last year's).
Can this be done with PHP or is the file way too large.

[Weedpacket]

PHP doesn't care how big the file is. Limitations will be on how much resource your system can give PHP.

[cyberlew15]

I have a Mini cluster of 4 2ghz pc's could i sort though a 1.2gigabyte rdf file or would it be easier to use a compiled language to interact with my server also how whould i get php to execute an exe i know how to get the data afterwards but i need php to execute the exe

[dewen]

why write the whole lof file into array for analyze? It could use up your resources. How you going to do with a array with millions of lines? I mean, there can be another way than array