dealing with Large files

newbie271

I am not sure whether I am at right place for the question or not. But someone might have an idea regarding this.

I have around 10 files each around 100 MB. I am developing a tool that uses some of the unix commands and external program such as C. I am testing my program on local host. To accomplish my job I need to bring them into memory and perform the required task.

I am worried that when this tool will be deployed and multiple users will be accessing the the tool simultaneously then how those file will be loaded. Will all be loaded separately for each user?

For example, if 5 people are accessing the tool, and suppose each tool is using all 10 files, will the same file be loaded 5 times in the memory at different location??

I don't know how php handle this? Can someone please please explain what happens in this case. Users are accessing the same file but the sessions are different.

Is it possible to load all the files in server memory and every user that uses the tool can have access to that part of memory so that the loading of the files does not take place separately.

I will be thankful for any suggestion. Or at least to point me to some website/forum/..or anything that can help me clarify this thing.

Thanks php users.

bradgrafelman

newbie271 wrote:
To accomplish my job I need to bring them into memory and perform the required task.

What 'job' is that? Do you really need all of the file(s) to be loaded into memory at once, or can you perhaps process them in a piecewise fashion (e.g. read in X entries/lines, process them, and then repeat that process until you've consumed the entire file)?

newbie271 wrote:
I am worried that when this tool will be deployed and multiple users will be accessing the the tool simultaneously then how those file will be loaded. Will all be loaded separately for each user?

PHP has the option of using shared memory ([man]shmop[/man] might be a good starting place). Otherwise, no, separate requests will invoke separate instances of the PHP interpreter which will have no correlation between any other instances.

newbie271 wrote:
For example, if 5 people are accessing the tool, and suppose each tool is using all 10 files, will the same file be loaded 5 times in the memory at different location??

Doesn't that make sense? If you start a session and store my name in $_SESSION['user_name'], does that mean every person who visits the site sees my name? Of course not - each PHP instance is separate from any other.

newbie271 wrote:
I don't know how php handle this?

The same way any other program would behave (be it a C program, a Unix shell, etc.). New instance invoked == new memory space used.

newbie271 wrote:
Is it possible to load all the files in server memory and every user that uses the tool can have access to that part of memory so that the loading of the files does not take place separately.

Yup; as mentioned above, the term you'll want to do some Google'ing on is 'shared memory'.

newbie271

hi bradgrafelman
Thanks a lot for your reply as well as pointing me to shmop functions.

I am trying to simplify what I am doing. You may have more insight into these type of problems.

I have 10 files and I am constructing tree from each of those files using C. C is doing all my work efficiently such as querying as compared to other methods such as mysql.
This is a kind of alternate way of using mysql. The only problem is that the tree needs to be in memory all the time for processing. The working code that I have is a simple php code and not using any advanced concept. I am assuming that the tool must be loading those 10 files for every user that accesses my tool which is a waste of time for the user and space on my server.

Since the files (trees) that are being queried are same for each user (only reading the files, no writing performed on these files) I was looking for some shared memory or some other concept through which these files remain in memory all the time and all instances of Php somehow use those files and not load them again. This will speed up my tool to a grater extent as loading of the files is the slowest step in my tool.

I hope I am able to explain what I want to do.

I am repeating my quesion.
Is it possible in PHP to load those 10 files once and every instance of Php accesses the same 10 files without separately loading those file for each instance.

Thank again bradgrafelman.

bradgrafelman

newbie271 wrote:
Is it possible in PHP to load those 10 files once and every instance of Php accesses the same 10 files without separately loading those file for each instance.

Didn't I already answer that question?...

bradgrafelman;10985161 wrote:
PHP has the option of using shared memory ([man]shmop[/man] might be a good starting place).

bradgrafelman;10985161 wrote:
Yup; as mentioned above, the term you'll want to do some Google'ing on is 'shared memory'.

leatherback

It sounds like you might actually be better of using a database.

newbie271

Hi bradgrafelman
Thanks a lot.
I just wanted to be double sure. I am trying to read some manual, mailing list regarding .
shmop. I have not come up with any code untill now.

leatherback,
Thanks a lot for ur reply. But for efficiency reason I am using this. I compared the mysql implementation and it is too slow.

Thanks again