Parsing HUGE file... is this possible?

jvanv8

I have a text file that is over 25mb in size.

My script reads in the file into a string, then breaks it apart into a 2 dimensional array then performs various operations (lots of for and while loops) on the array such as inserting data into a mysql database.

The mulidimensional arrays may contain 30,000 columns which each column holding an array of 30 attributes. ( these are just estimates)

Will this be possible with such a huge file? I ran it a few times and it failed to read in the entire file contents.

What precautions should I take? I reset the apache timeout - are their others I should take note of (PS. I'm on a mac, can't find the php.ini file).

Are there array size limits I should know about?

Thanks
John

mrhappiness

does it fail to read the file entirely or does it fail processing the lines?

I'ld suggest splitting up the array, e. g. processing the first 1,000 lines of the file, then doing

header('location: process_file.php?start=1000');

and processing the next 1,000 lines

jvanv8

Yup, that was my fallback alternative solution.
Always nice when you don't have to resort to that sort of stuff though but I probably don't have any choice here.

However, the script wasn't even reading the entire data from the text file. I was thinking of maybe breaking the text file into 2 or more files and read those. Is there a way to automate that?

mrhappiness

maybe something like this will work

$fp=fopen('largefile.txt','rb');
$line=0;
$length=(isset($_GET['length'])) ? (int)$GET['length'] : 0;
$content=array();
fseek($fp, $length);
while (!feof($fp) and $line++<1000)
  $content[]=fgets($fp);
$length+=strlen(implode("\n", $content));
process_array($content);
header('location: '.$_SERVER['HTTP_HOST'].'/'.$_SERVER['PHP_SELF'].'?length='.$length);

without any warranty, just a quickshot

jvanv8

Thanks! I'm probably going to have to think of a better way of "splicing" the file instead of using lines. The file does not contain any newline characters and fields are separated by tabs.

The problem is that the file should only be cut exactly where a new top level array begins. There isn't a "flag" to determin a new array value. Since each top level array entry has about 30 tabs (each value in between makes up the 2nd array) I might be able to detect occurances of a TAB (as the number N) and find the
(round(N/30))/2)*30
occurance as a general "half way" point where it's safe to cut

I think this would prevent cutting the file in 2 halfs with half of an array in one file and half of an array in another file.

Of course, this assumes that php can count the number of total tabs in the document.

John

mrhappiness

it would be easier to change the format of the file if possible

you can only count the tabs in the file if you read it entirely

jvanv8

unfortunately, I have no control over the contents of the file.

Maybe instead I could just cut the content with associated with first 300000 tabs and place it in a new file... and continue doing so until an end of file is triggered...

lazzerous

well, you wouldnt want to alter php.ini, as it would affect all dirs and files

However, you could use .htaccess to override php.ini settings on a directory-specific basis

just remember - .htaccess affects all contents of the dir it is placed in, including all sub directories and their respective contents.

-=Lazzerous=-