splitting a large file

Anon

I'd like to split some files, because they are too large for me to analyse them(100Megs and above)
See how i do it:

$file = "file";
$fp = @fopen ("W3SVC36new.log","r");

$long=10000000; //size of the little files
// created

$i=1; //this is to create each time

a new file name

while (!(feof($fp)))
{
$fp2 = @fopen ($file.$i,"w");
$tableau = @fread($fp,$long);

$table = explode("\n",$tableau);
foreach ($table as $elem)
{
fwrite ($fp2,$elem);
fwrite ($fp2,"\n");
}
$i=$i+1;
}

In fact, i specify a size in the fread, i format the string i get, and i store it in a new file.
I do this until the end of the Large File.
The problem is that, using fread, i often cut a line in the middle, so a part is in a file, and the end is in the next file created.

I hope someone will understand and help me.

Thank You.

Anon

Hrmm.. interesting problem.. lets look at it this way.

Although it may take a couple more seconds, your best guess is the following:

$file = "file";
$fp = @fopen ("W3SVC36new.log","r");
$long=10000000; //max size of the little files created
$i=1; //this is to create each time a new file name

while (!feof($fp)) { // Go through the while file
while ((strlen($buffer) < $long)) { // Grab a chunck that is approx size of $long (don't cut lines)
$buffer.= @fgets($fp,2000);

}
$fp2 = fopen($file.$i, "w"); // Go ahead and write the buffer we just grabbed
fwrite($fp2, $buffer);
fclose($fp2);
$buffer = ""; // Reset buffer for next go'round
$i++; // Increment counter
} // Continue loop
fclose($fp);

That should do what you are looking for 🙂 Hope that helps

Chris King

[deleted]

There are tools for doing this.. search the net!

If you are on unix, you can use "split"

spider_man

First of all overrid the default 30 sec. time limit for the script (It will take longer than that):

ignore_user_abort(false);
set_time_limit(0);

Then split the file by newlines:

$log = file('log.txt');

$i = 0;
$chunk = '';

while (list(, $value) = each($log))
{
if ((strlen($chunk[$i]) + strlen(next($log))) > 10000000)
$i++;
$chunk[$i] .= $value;
}

unset($log);

Now you've got the small files in the $chunk array.
Haven't tested it but it should work fine.

Anon

Here's what I'd do, although I can't spare the time just now to work out the complete PHP syntax!

Use this to get a big chunk of the file
$tab = @fread($fp,$long);

then use this to add the small chunk up to the next newline
$tab .= @fgets($fp,1000);

now write that lot to the output file
$fp2 = @fopen ($file.$i,"w");
fwrite ($fp2,$tab);
fclose ($fp2);
$i++;

... and keep doing that until you get to the end of the input file.

I assume you're using PHP because this big file is remote to you on your ISP's server or whatever. Otherwise, like Vincent says, I wouldnt be using PHP to do this.

spider_man

Guess you come from Perl.

But how do you cut it by the line breaks?

[deleted]

reading 100MB line by line.... wow...

Liek I said, this is a job for a binary, not php :-)

Anon

You gave me a lot of solutions, and, for that, i thank you all.I haven't tested any yet, but i'm sure they'll work.
And yes, Vincent, i agree with you "This is not a job for php" But the problem is that i am a newbie in php, so i have to practice as much as possible, AND this "BIG FILES" problem had to be be solved, so...