[RESOLVED] can't read contents of large file - times out

[ppowell]

if (!function_exists('bigfile')) {
 /**
   * Works like file() in PHP except that it will work more efficiently with very large files
   *
   * @access public
   * @param mixed $fullFilePath
   * @return array $lineArray
   * @see actual_path
   */
 function bigfile($fullFilePath) {
 @ini_set('memory_limit', (int)ini_get('memory_limit') * 10 . 'M'); // MULTIPLY YOUR MEMORY TEMPORARILY (10X LARGER SHOULD *DEFINITELY* BE ENOUGH UNTIL END!)
 $fileID = @fopen(actual_path($fullFilePath), 'r');
 while (@!feof($fileID)) {
     $buffer = @fgets($fileID, 4096);
     $lineArray[] = $buffer;
 }
 @fclose($fileID);
 return $lineArray;
 }
}

I even temporarily increase memory (I know, bad idea but it's all I can think of to do), however, requirements stipulate that files that are smaller than the max file size (arbitrarily set) be sent via email attachment (of course, depending on email SMTP server if it gets sent)

I can't think of any other trick to make this either work or NOT to time out but throw an error/warning.

Help!

Thanx
Phil

[etully]

How big are the files where it fails? (This way, I can run a similar test on my machine).

Before I test, let me ask, is it possible to process the file one line at a time instead of putting the entire file into an array? For example, if you are outputting the file to the screen, just print or echo the data instead of storing it in an array. If you are scanning the text looking for some value, only add the line to the array if the line is going to be useful or meaningful later. If you are converting the file in some way, make the conversion on that one line and then write out that line (either to the screen or another text file).

I had this problem once on very large files (10-20M😎 and it was taking 20-30 minutes to process. What I did was this:

1. Set the max execution time to 30 mins
2. Write a message to a table that said "processing"
3. Print a meta refresh on the page that forced the browser to jump to a second page.
4. Start processing the large file
5. When done, update the database table to say "done"

Then, in the 2nd file, I did this:
1. Check the database table to see if it said "done"
2. If yes, then print a meta refresh to the 3rd and final page that said "Done!!"
3. If no, then print a 5 second meta refresh to the 2nd page

[ysu]

hm...how about reading the file size prior reading the whole file in, and throwing the error message if it's over a set limit?

[ppowell]

ysu: I do that already, but this is a portable web application, so there is no way of ever knowing what the maximum file size limit can be since it's never the same for all machines.

etully: Thanx for your advice, however, what I am doing with this file, in this case, is using it as an attachment to an auto-generated email. I found a way of doing that now by writing my own function that mimics the actions of file() while allowing for handling of large files, calling it bigfile():

if (!function_exists('bigfile')) {
 /**
   * Works like file() in PHP except that it will work more efficiently with very large files
   *
   * @access public
   * @param mixed $fullFilePath
   * @return array $lineArray
   * @see actual_path
   * @link http://us2.php.net/manual/en/function.file.php#30503
   * @see link regarding usage of bigfile()
   */
 function bigfile($fullFilePath) {
	@ini_set('memory_limit', (int)ini_get('memory_limit') * 10 . 'M'); // MULTIPLY YOUR MEMORY TEMPORARILY (10X LARGER SHOULD *DEFINITELY* BE ENOUGH UNTIL END!)
	$fileID = @fopen(actual_path($fullFilePath), 'r');
	while (@!feof($fileID)) {
   	 $buffer = @fgets($fileID, 4096);
   	 $lineArray[] = $buffer;
	}
	@fclose($fileID);
	return $lineArray;
 }
}