PHP4 process memory size

[Anon]

Guys,

I have an issue with the memory used by Apache/PHP on some of my servers. With large applications (meaning lots of PHP source code) .... the size of the process can grow very large. PHP4 does not seem to be releasing the memory used, it looks like the source s being parsed and cached within Apache/PHP processes.

Has anyone else noticed this? It could be quite a big problem for any complex application, as the ratio of process size to source code varys between 10:1 and even 50:1.

I've created some sample PHP source files, in the region of 900K, and included them in a simple test script. Access this test script caused the process size to grow from 7MB to 55MB.

My environment is:

Linux (Red Hat v6),
PHP 4.0.3 r1. compiled with:
./configure --with-apache=../apache_1.3.14_php4_oci8 --enable-track-vars --with-config-file-path=/tmp/ -without-gd --with-oci8

Server version: Apache/1.3.14 (Unix)
Server compiled with....
-D HAVE_MMAP
-D HAVE_SHMGET
-D USE_SHMGET_SCOREBOARD
-D USE_MMAP_FILES
-D USE_FCNTL_SERIALIZED_ACCEPT

I've seen similar problems on Solaris 2.6

Also the raw size of the process, before accessing pages is 2MB configured without OCI8, and 7BM with OCI8, is there any way to cinfigure with shared Oracle libraries to reduce this 7MB overhead?

thanks,
Ronan

[Anon]

Are you keeping track of shared memory usage versus non-shared memory usage?

Generally speaking, shared memory usage is a good thing, since it means that the code you just ran can be run again quickly without having to grab it from the hard drive and recompile it to byte code all over again.

The real test is how the server behaves under higher load than just one user. Crank up ab against the script with a -c 10 or 20 to get a feel for how the server uses memory with a lot of people accessing the same large file at once. Generally, the memory usage will not go up by a factor of one for each added process, but something much smaller like 10 or 20% max, since the code in use is in shared memory.

[Anon]

I agree that shared memory is better than memory owned exclusively by a
single process, but what I am concerned about is that PHP4 seems to use
a huge amount of memory, based on the amount of source code.

A ratio of 50 times (or even 10) memory to raw source code seems very
inefficient. I would have expected something like an inverse
relationship (e.g. 100K of source code requiring 25K of memory once
parsed), before any variable storage or arrays are considered.

What I am hoping is that someone can point to a compile option, or a
patch/work in progress that might reduce this memory overhead.

[Anon]

First point, is that PHP will not "release" memory it is using for buffers unless the machine gets low on memory, and why should it? It's faster to hold the last few pages in memory than to just flush them and hope nobody else wants to see it again.

Second point is that PHP memory usage is still tiny compared to say cold fusion or some of the big servers like websphere, where each child thread can weigh in at 25+ Megs at idle, with no cache or buffer memory used.

Third point is that until you test the machine under some kind of realistic mode, you won't see how it really behaves. My experience is that as simultaneous users increases, apache/php allocates less and less memory per process for buffer memory, so that total memory usage does NOT go up by simple factors for the number of users. i.e. the shared memory used does not grow at a linear rate with an increase in the number of users.

Try running your script via ab with a line like this, while running top, and see what numbers you get:

ab -n 100 -c 5 http:yourserver/script.php

-n is number of accesses, -c is conncurrent connects.

[Anon]

Scott Marlowe wrote:

First point, is that PHP will not "release" memory it is using for buffers unless the machine gets low on memory, and why should it? It's faster to hold the last few pages in memory than to just flush them and hope nobody else wants to see it again.

-> Do you mean that there is no memory management, that if you have lots of scripts running, with lots of users, that the memory usgae will just grow and grow?

Second point is that PHP memory usage is still tiny compared to say cold fusion or some of the big servers like websphere, where each child thread can weigh in at 25+ Megs at idle, with no cache or buffer memory used.

-> My point is that the child processes are in the 25MB to 50MB ranges, which I feel is excessive, and I am wondering is there something that can be done to prevent this. Try generating a simple script with 10,000 lines of code, and watch the memory used, even if this code is never executed!

Third point is that until you test the machine under some kind of realistic mode, you won't see how it really behaves. My experience is that as simultaneous users increases, apache/php allocates less and less memory per process for buffer memory, so that total memory usage does NOT go up by simple factors for the number of users. i.e. the shared memory used does not grow at a linear rate with an increase in the number of users.

-> I have, I've ran my application with the ab, and with 5 to 10 real people as testers. Bothe show memory growing on every process used (we would often configure a server with 50 to 100 apache processes, and over 1GB of ram. But even this 1 GB ram gets used very quickly, the machine swaps, ... performance dies off.

[Anon]

There is memory management, it's just that as long as there's memory left, PHP/Apache will tend to use it with abandon, as will many databases out there.

900k is a HUGE script file. If your files are this big, you need to point the zend optimizer at it.

Finally, if you run a phpinfo(); command from a script, do you have --enable-inline-optimization turned on? (it'll be listed at the top with the ./configure switches) This tends to use memory much faster, but gives better performance.

[Anon]

We do have the Zend optimizer configured and in use, it certainly helps the performance but it does not make the process sizes bigger
or smaller.

We do have a big application, its of the order of 80,000 lines of code, contains a lot of classes all of which are used. I can streamline
the code to some extent so that only half of the code needs to be loaded at any one time, however there is still an issue with the amount
of space used. I do not understand why loading 900K of source should use more than 25,000K of ram. It looks and I think is very
inefficient.

I have seen, with PHP4, both on Linux and Solaris, cause powerful servers to run of memory because of this issue. I've checked the
inline configure option and its not enabled.

For large scale applications 80,000 lines of code is not that strange, we have seen and developed several web applications of this size
in different languages, including PHP3 without memory issues. If you think of 50 Apache/PHP4 processes using 20MB each, that
requires 1GB of RAM (before counting variables and temporary data stacks etc., and presuming you do not want swapping).

I know that executables on Windows machines are often 20MB+, but I think that these contain a lot of extra DLLs and code which is
rarely if ever used. In general I would expect the relationship between the internal machine representation (compiled code, elf, a.out,
bytecode, machine code or whatever) to be smaller than the source code (excluding data stacks for arrays etc.). Everything else about
PHP4 and the Zend engine looks very efficient, so I really am surprised at this. Surely there must be some way to reduce this
overhead? Am I the only person to notice this, if so could be caused by different libc library versions or something else unique to my
servers?

[Anon]

While I agree that for a large application 80,000 lines of code is no big deal, having all that code (or even a large chunk of it) in one big file is a poor design decision.

So, do you have a single page that actually needs that big of a file behind it, or is any one page only using a tiny percentage of that huge file?

I would definitely look at partitioning your code into smaller chunks, and only includeing the ones that a certain page needs to reduce memory usage.

The largest file we have on our php server is a 350k file used for printing out the company phonebook, and it is a huge, hideous abomination of code that should be rewritten with many more function calls and common code cut out I could get it down to a 100k or so.

On my machine that file makes the normally 4/2 (abs/shared) Meg apache/php children grow to 6/3 Meg children when running. It has about 4,000 lines of code in it, if memory serves.

While I can see thousands of lines of code being used for a large app, I can't see them all needing to be loaded at one time for a single page view, so I have to wonder how you've approached the segmenting of your application if php files or 300k and up are getting loaded very often.

Also, are you doing a lot of logic in your code you could move off to your database via stored procedures, triggers and such?

[Anon]

With 80,000 lines of code, how much of the output that you create for a user each time is unique and non-cacheable? In other words, how often does all of the output change, and how big is that output?

I've created a few nightmare programs because of a lack of available time to code it right, it just had to be a big, ugly database beating peace of code, and it took over 30 seconds to run on my older server. While it's down to under 10 on the new one, it's such a pig that it's cached because it only changes every night. Even if it changed every 5 minutes, or even every 20 seconds, it would still be cheaper to harvest it every 20 second and only serve the static page for performance.

On the other hand, if this thing creates a unique output EVERY time, in every respect, that can't be approximated, you may be stuck just using php3 unless there's some part of 4 that you need. If php4 handles sessions, you could still let it do that, and run php3 boxes behind that/apache as one task robots that did nothing but run this script. If PHP is really that much better at memory allocation, and I'm not doubting it is, then that might be the best scaling solution. If you don't need php4 you can just load balance php3 servers behind a cisco or linux router, if you even need more than one.

[Anon]

To a large extent, nearly all the output is dynamic, it does have quite involved user sessions, although these are coded to work in PHP3
or PHP4. Some of the application requires a lot of pattern matching and so is also very CPU intensive, which is why we moved to
PHP4 - Zend really is streets ahead in terms of performance.

And, yes, we can certainly partition the application, it is well structured, with many classes spread over 40 or 50 include files. All static
HTML parts are already in raw HTMl in headers/footers etc. It was originally coded in PHP3, which at the time did not support
complex/multiple includes as well as PHP4 does now, and so was modeled as a single file (which acted as like an event loop in a
windows program). In this way it was easy to ensure that all files were included only once. So, I'm sure we can cut the required code
down to a max of 40% of the total, or there about.

I'm just aiming to attack all the angles and see if we also work out why PHP4/Zend needs so much memory for its "compiled" code.
Perhaps Zend cache or other products will address this issue, I feel it will need to be addressed for large scale application
development. For these type of large projects with many developers there will always be trade-offs between application design and
"project design"!

Scott - Thanks for your help to date, it has been useful. I think my next line is to post a few queries to the Zend team, and ask about
some of the future products.