Memory Problem

Anon

I have to read and test several files, but most of them are quite large (40-50 Megs)
So when I start the script that is supposed to test those files, after a while, my script stops, because "Windows hasn't enough virtual memory", or something like this.
Is there a way to clear memory each time I open a new file, or each #Line of those files?
I you have the answer, please help me.
Thank You.

[deleted]

How do you read the files now?
Do you just fread($fp,40megs);?

I would read the file line by line or 10k at a time,
that way every time you read new data, it overwrites the old data, saving memory.

Anon

I used the command Dir(name_of_the_file), that is supposed to store the entire file in an array.
That's quite raw, i must admit.
I thank you for your advice, i'll try the fgets command.That will probably be slower, but it has more chances to work.
By the way i'd like to apologize: if you have problems to understand what i say, it's because i'm french.And a mere french computer-scientist like me doesn't have much vocabulary.
Sorry.

[deleted]

You probably meant you used "file()" :-)

And that does indeed read the whole file into memory.

Ah, so you are french.. oh well, nobody's perfect! :-)

Anon

Yes, i used File() indeed.
Dir was just used to make a difference between files and directories, and to explore them all.
My problem now is that the line by line method is much to slow (as i knew it would..)
maybe i should mix both file by file and line by line method.
By the way....saying that nobody's perfect, you meant that you are english?

[deleted]

If you are reading a binary file (guess not because you were reading the file into an array) you might speed things up by reading 10k at a time using fread.

Nope I'm not english, I'm a Dutchie, which is allmost perfect, I know. :-)

babak

What has file() to do with nationality?
😉
//babak (Fisrt generation Iranian in Denmark)

Anon

I've already tried fread.The problem is that fread can't store data in an array (or at least I didn't find how to)It seems like Fread simply returns a string.
So how will I separate each line of these huge files?

Dutchie eh?
Well I guess perfection will require much more modesty, Dutchman.
(And maybe more vocabulary too?
Allmost->Wrong
Almost->right)
I'm not perfect, and it's the best way to improve myself

Don't get hurt, please I had to tell you this.
I hope you will help me, despite what I said.

Anon

Actually, nothing.I was just trying to be modest (because French people aren't known to be...on the contrary!)And vincent criticized me.That's nothing, really.
Could you help me, Babak the First?

babak

Do you know the size of the file you are trying to read?

Anon

Actually there are many files some have 10Kb, and some have more than 50Megs...
A file by file method is fast, but requires too much memory.
I think I have to use Fread, BUT in that case i have to find a way to store the string
that fread returns in an array, each line of the array corresponding to a line in the file
I hope i am clear enough.
So, if you have any idea...anyway, thank you.

babak

We have experienced with reading various files and monitoring the memory (LINUX), and the result was amazing.
Twice as size of the file was consumed to read the file.
Your problem is probably caused by the capacity of the variable you are using to store the file content in.
I don't think that either the physical nor virtual memory runs dry because of that.
Can you please tell what you like to do with the file?

Anon

For each file, i have to find a particular string, (#Fields, to be precise) included in a line .Each time I find it
Two Possibilities:

1)The line is different from the ones I have already encountered:
in that case, i diplay this new line, i display where it is in the file, and i give it a number to identify it

2)The line has already been encountered:
in that case, i simply display its position, and the number to identify it.

Anon

After processing a file, you should use unset() to free memory. I'm not an expert in PHP memory management (I'm French too, so I'm not perfect neither :-)), but I think that memory is never released until the end of a script unless you use unset (if some people have more info about this, I'll be glad to know). So your code should look like this :

void ProcessFile($filename)
{
$a=file($filename);
$nb=count($a);
for($i=0;$i<$nb;$i++)
{
//Your processing
}
//Free array
unset($a);
}

Maybe it will solve your problem.

P.S. Your English is very nice !

Anon

Thank you Olivier, for your advice, but i already tried Unset to free my array's memory. But it has no effect.
Actually, I thought about using fread, to get only a part of one of these big files.
The problem is that i won't be able, with fread, to make a difference between each line.(because i have to analyse these files line by line)
If someone know a way to get only a part of a file BUT keeping a way to make a difference between each line, let me know.
Olivier, thank you for your support.
"je suis content de savoir que mon anglais est compréhensible par un français, c'est toujours ça de pris"

babak

Hello again
I've discussed this problem with some really experienced PHP'ists, and they all said, that it doesn't sound as a good idea.
What jind of files are you trying to read anyway?

They point is that it is very demanding to read 50MB at a time, and store it for later processing. You better read few byte and do the job with another solution.
//babak

Anon

This is precisely what i have in mind.
I'll separate the "Big Files" in smaller parts.
I'll use Fread, and in the string i get, I'll use Ereg to identify the specific string i'm searching.Now that i know to specify how to separate each line, this should work.