The best thing, of course, would be to meter the code and the site to determine precisely where the bottlenecks are, and then determine the best means to lessen them. That being said, I would think that as a general rule of thumb you would be better off with a completely separate server for the database. Not only does it then have its own processor, but it has its own disk drive(s) for which it does not have to compete with the web server, PHP, and any other applications to access. But I am by no means a performance guru, so just take the above as "food for thought".

NogDog, you nailed that answer. I came in to write precisely the same thing.

You can never just assume that CPU is the cause of slowness. It can be RAM, Database, Hard Drive, Bandwidth, CPU, inefficient programming, or a combination of those factors. If you just guess that CPU is the problem, you have a one in seven chance of being right.

You guys are totally right...I should have been more specific in my question. Let me rephrase it:

Given that a faster server is probably cheaper than me totally re-examining my code (extra cost for upgrade to dual-core AMD 3800+ is about $720 per year), can I be sure of a performance gain for Apache/PHP/MySQL when I move to dual-core system? Basically, does a LAMP system really take advantage of a multi core/multi-proc system or will apache and mysql (the CPU-munching procs on my machine) still suffer from sequential execution?

More info: (also check here)
My server is a 2GHZ Celeron processor with 768 MB running Fedora Linux. The kernel is a bit old because it's been running for awhile (463 days since reboot!):

Linux myplan.myplan.com 2.4.22-1.2115.nptl #1 Wed Oct 29 15:42:51 EST 2003 i686 i686 i386 GNU/Linux

.

RAM
I know my server needs a little more RAM. The free command during peak hours indicates that there's only about 20MB of RAM available and about 120 MB of disk swap are in use. However, I don't get the feeling that adding a a few GB of RAM is really going to provide the performance capability to address our traffic growth moving forward.

DISK
As far as I can tell, the system is NOT I/O bound. The top command reveals the CPU usage rocketing to as high as 97% in short bursts. I have never seen the 'iowait' value in the top output be anything but 0.0%. Given that most queries to the db return only small amounts of data, I'm pretty sure that the disk drive speed is not at issue.

DB
As I just said, nearly all queries return only small amounts of information. I spent a lot of time working on my slower queries to try and optimize the data structures. Most of the queries that are not mine are from PHPBB. I'm CERTAIN some could be improved, but am wondering if it's worth my time and effort if I can get a significant boost from a $700 investment in server hardware.

BANDWIDTH
Hm. Hadn't yet considered that. The site is relatively image-intensive in its layout. Any quick tips on how to gauge that quickly?

CPU
As described above, the CPU frequently bursts into fairly high percentage ranges even during low to moderate traffic. The load average is typically 0.65 or higher during moderate times and I haven't yet been able to log on during peak usage to find how it's working then.

INEFFICIENT PROGRAMMING
HA! FAT CHANCE. But seriously I'm sure there could be performance improvements. For instance, I include a lot of functions in my init.php file for convenience that are not used on every page. I tend to be pretty attentive to inefficient situations though and preliminary performance profiling indicated that db queries and a few other select routines were orders of magnitude slower than the convenience shortcuts I took.

My admittedly cursory analysis says that, aside from poor programming or DB construction, RAM and CPU are the culprits. At the very least it would seem wise to test the very affordable option of a faster machine before totally reworking my code.

If there's some kind of awesome tool that could point out my bottleneck here, please let me know what it is!

You'd don't have to do much to the PHPBB database. Just have a look at the indexing and see if obvious things like dates are indexed, make a note of what you change and see if there is anu improvement. Re-apply those changes on update.

Only takes 1 index to drag things down and I did read wordpress forgot to put an index on a date at one point causing it to drag as it got bigger.

Have you run even quick tests to determine what might be going on? For example, what is the load time of your scripts? Does this vary on pages with more/less database queries?

How many database queries do you have per page? How do pages w/more queries influence load time?

How many includes do you have per page? I more than tripled the speed of my CMS/Framework by reducing this.

Have you tried a profiler to see where your script is going slow? Sometimes you'll find you're calling the same functions multiple times (accidentally of course).

Have you tried any accelerators? I've been using eAccelerator and have had significant speed improvements since using it. Requires no changes to your code at all, and relatively easy to install.

Are you using any sort of file/query caching? Load time per page decreased from 0.03 - 0.06seconds/page down to 0.0001seconds. I also use query and object caching for more resource-intensive operations.

Have a closer look at your database structure, indexes, and queries. Can they be optimized/improved? Run some of them in PHPMyAdmin and see how long they are taking.

If PHP is your bottleneck, increasing CPU power will help. If it's MySQL, then increasing the ram should help alleviate the strain.

I appreciate all this well-intentioned advice, but I sort of just wish someone would answer my question (about whether or not a multicore cpu really makes a difference on a machine running Apache 2, PHP 4.3.8, and MySQL 3.23.58).

On the subject of multithreading - PHP is thread safe, but not itself multithreaded. I believe there are some extensions which aren't thread safe. Also correct me if I'm wrong but Apache 2 should be run in it's prefork mode when using PHP, so is also not multithreaded.

Shrike wrote:

On the subject of multithreading - PHP is thread safe, but not itself multithreaded. I believe there are some extensions which aren't thread safe. Also correct me if I'm wrong but Apache 2 should be run in it's prefork mode when using PHP, so is also not multithreaded.

The warning: http://www.php.net/manual/en/faq.installation.php#faq.installation.apache2

From that, it might be OK to run Apache 2 in MPM mode if on a Windows server or using the fast-cgi PHP on a *nix server.

Also, you might want to look at this article about a sort of pseudo-threading with PHP: http://www.alternateinterior.com/2007/05/multi-threading-strategies-in-php.html

I've found reasonable success with threaded Apache and mod_php, this means that PHP runs truly multithreaded. Some versions are better than others - one I ran for months with no problems, others suffered from frequent crashes.

fastcgi would alleviate it though, by running a pool of PHP processes.

Generally speaking, running in prefork Apache is very inefficient for memory, as you will have to set MaxClients high enough for your static file load (images, etc), but those processes still use up ram with PHP's junk - which is immensely wasteful. Multithreaded is much better, fastcgi is probably also ok, but there are other issue with that I guess (I've not tried it).

As with anything, the trick is to simulate load on your lab bench, turn it up higher and higher and monitor performance - you can then compare different configurations on your lab bench before going to production.

Multi core CPUs are cheap - I'm sure you can afford a few boxes for your lab.

Mark

cgraz: That's a pretty good checklist when I get around to re-examining my code. The bottom line is that I expect to exceed my current machine regardless of code improvement or not very very soon. the client has authorized machine upgrade and i expect to cash in!

shrike: god bless you friend. that is precisely the line of discussion i hoped to open. am i mistaken in thinking that threadsafe is useless unless the parent process (in this case apache isn't it?) can spawn multiple threads? Am i right in thinking that apache 2 might do this? or might not? does it depend on compile? is it safe?

nog tha dawg: YES that warning is ominous isn't it? also, IMHO it's also a complete pussy-ass cop-out on the very important issue of multithreading. I realize that PHP's success and popularity are due to this 'glue' aspect, but PHP must evolve or we are all on a sinking ship. Your other comments are a bit beyond me...gonna have to read up to understand.

MarkR: Lab? I work at home. I'd send you a picture if it wasn't so sad. And rent is late. I do have access to a multicore cpu tho. Any ideas for a simple test to determine how much concurrency a LAMP stack generates? I can think of a few things to test: Will php cycles and mysql cycles be executed concurrently on separate cores? can apache spawn multiple instances of php to the various cores for concurrent execution? what about other system processes like crond or spamd? do these get farmed out to the other cores?

I guess what I'm really trying to understand is whether there's a way to configure a LAMP stack so that it really takes advantage of multicore systems. The best scenario would obviously be where mysql can run multithreaded and the system can spawn a few differerent concurrent PHP execution threads off to the various cores. MarkR pointed out the obvious issue of memory usage being a potential pitfall.

What about safety? That warning nogdog put up exists in a longer version somewhere which rightly points out that concurrency problems are not always evident in a given system -- they usually manifest in larger scale systems and result in utterly unpredictable consequences. They tend to be caused by one thread overwriting the memory of another so you get these bizarre results which may not be caught by the usual exception handling of an application.

EDIT:
In reading up on MySQL optimization yet again, I found this bit of happy news which says:

The operating system to use is very important. To get the best use of multiple-CPU machines, you should use Solaris (because its threads implementation works well) or Linux (because the 2.4 and later kernels have good SMP support).

This would at least imply that even older versions of mysql can take advantage of multi-cpu or multicore systems. In the end, if I could get apache/php executing on one core and mysql executing on the other I would consider that a really happy thing. I'm not convinced this would actually happen, however. I suppose MarkR is right in saying we should devise some kind of test. Benchmarking experiment anyone?

Ok, so beg, borrow or steal (actually, probably better not to do the last one) a multi-core cpu.

Performance-test your application with both cores enabled and with one core disabled, to see what kind of performance benefit you get.

I'm reasonably convinced that if you're simulating multiple simultaneous users, you'll get a good benefit unless:

• There isn't enough RAM
• The machine is already I/O bound
• All your threads are doing operations most of the time which need to be synchronised.

The last point is interesting. MySQL has various techniques for preventing two threads from screwing up a table - including table locking, row locking etc.

The upshot of this is, that certain operations can block other operations out, meaning that even if there are spare CPUs to run them on, they can't proceed until a previous one is finished.

The MySQL contention issue is far to big to discuss here - feel free to peruse the chapters in the manual about how the engines deal with locking (hint: They don't all do it the same).

Mark

OK so I tried a little benchmark that was really encouraging. I begged my brother to borrow his server with two dual-core CPUs (yes, 4 cores) and I installed PHPBB on it. I then took another machine on the same network and ran ab, asking for the index page of phpBB 1,000 times.

So on the first server (the one with phpB, this mpstat command to capture CPU usage for each core:

server1:/home/mydomain/public# mpstat -P ALL 1 1000 > benchmark.txt

Then on the other server I started apache benchmark to hit server one:

server2:~# ab -c 10 -n 1000 http://mydomain.com/community/index.php

I'm not certain ab was doing what i wanted yesterday...the byte counts returned looked suspect. My phpbb page is 20,997 bytes and it was returning 282 bytes or something.

BUT TODAY I ran the benchmark test again and the output from 'ab' on server2 was:

server2:~# ab -c 10 -n 1000 http://mydomain.com/community/index.php
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking mydomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests


Server Software:        Apache/2.2.3
Server Hostname:        mydomain.com
Server Port:            80

Document Path:          /community/index.php
Document Length:        21000 bytes

Concurrency Level:      10
Time taken for tests:   92.944917 seconds
Complete requests:      1000
Failed requests:        297
   (Connect: 0, Length: 297, Exceptions: 0)
Write errors:           0
Total transferred:      21658109 bytes
HTML transferred:       20999109 bytes
Requests per second:    10.76 [#/sec] (mean)
Time per request:       929.449 [ms] (mean)
Time per request:       92.945 [ms] (mean, across all concurrent requests)
Transfer rate:          227.55 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   320  926 481.1    832    7198
Waiting:      307  899 466.1    802    7184
Total:        320  926 481.1    832    7198

Percentage of the requests served within a certain time (ms)
  50%    832
  66%    967
  75%   1057
  80%   1123
  90%   1383
  95%   1676
  98%   2168
  99%   2944
 100%   7198 (longest request)

I'm not sure that's all that great in terms of performance. 10 requests per second doesn't sound like a lot to me.

HOWEVER (and i think this is rather glorious) the CPU usage on all 4 cores averaged about 85% user and 15% system utilization. Unless I am misunderstanding something or unless the results of mpstat are misleading, It would appear that my server is using all 4 cores to serve up my page. mpstat results from server 1 (just a samping):

04:36:32 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
04:36:33 PM  all   84.97    0.00   14.61    0.00    0.00    0.21    0.00    0.21   1120.00
04:36:33 PM    0   89.08    0.00   10.08    0.00    0.00    0.84    0.00    0.00    121.67
04:36:33 PM    1   86.67    0.00   13.33    0.00    0.00    0.00    0.00    0.00      0.00
04:36:33 PM    2   83.47    0.00   15.70    0.00    0.00    0.83    0.00    0.00      0.00
04:36:33 PM    3   80.17    0.00   19.01    0.00    0.00    0.83    0.00    0.00      0.00

NOW it occurs to me...but this is apache 2 right? It's multithreaded, right? What if I had it in prefork mode? How do I tell if it's in worker or prefork mode?

But seriously, folks...how do you tell if your Apache 2 install is prefork or worker mode? I tried the command from the apache docs but it just doesn't work. I can't find where httpd lives in order to run the command './httpd -l'

I've also looked at phpinfo() results and I see under 'Loaded Modules' the truly encouraging word 'prefork'. Does that mean I'm in prefork mode?

Loaded Modules: core mod_log_config mod_logio prefork http_core mod_so mod_alias mod_auth_basic mod_authn_file mod_authz_default mod_authz_groupfile mod_authz_host mod_authz_user mod_autoindex mod_cgi mod_dir mod_env mod_include mod_mime mod_negotiation mod_php5 mod_rewrite mod_setenvif mod_ssl mod_status mod_suexec mod_userdir

Ok, firstly, Apache creates several threads regardless of whether you're in worker or prefork mode, but in prefork mode, each thread is in its own process.

Because it uses several threads, regardless of whether it's in prefork or worker, it can still benefit from multiple CPUs.

I don't think your testing results are meaningful, because you need to repeat them with some of the cores disabled on the same box.

Try disabling all except one of the cores (Hint: boot linux with the nosmp flag), then repeat your experiment, to compare the results.

Mark

MarkR wrote:

Try disabling all except one of the cores (Hint: boot linux with the nosmp flag), then repeat your experiment, to compare the results.
Mark

Where does one set the nosmp flag? Does this involve a BIOS change?

The "nosmp" flag is a Linux kernel command line option. Consult your distribution or bootloader's documentation to find out how to set it.

It's usually a boot-time option which can be set by interrupting the boot process and changing some options (temporarily; it will only affect the machine until reboot- then the other cores will be reactivated).

Consult your distribution's documentation or your systems manager.

Mark

To get back to the original, very valid question:

Is it worth the money for a dual-core CPU to host an Apache/PHP web server?

NO. It's not. Apache running in pre-fork mode (the default setting with Apache 2 and the only safe way to run PHP) won't take advantage of the dual cores. You're better off with a 3.0 ghz single core than a 2.0 ghz dual core. Your PHP processes -- which account for most of the CPU activity -- can't take advantage of both cores.

What if you're running MySQL on the same server?
Yes, you'll benefit from the dual cores. MySQL is mult-threaded and will utilize both cores without your changing a thing.

How can I speed my LAMP application on a multi-core system?
Upgrade to MySQL 5 and look for places to move PHP database recordset handling or logic into MySQL stored procedures. Note, you don't have to do any fancy multi-threading coding. Since MySQL itself is multi-threaded it automatically can make use of both cores anytime its dealing with concurrent query requests from multiple clients.

I think you are mistaken regarding apache/php. Prefork Apache does create a number of separate processes before they are actually needed as described here. The main differences between this and true multithreading are a) they are created ahead of time, not on deminad and b) each process is in its own memory space so PHP's non-multithreaded construction should not matter--or at least that's how I understand it. I'm currently wondering what happens if you try to write a file or image or something with your php script - could you have race conditions or deadlock in two separate apache/php processes or does linux take care of that?

I am running MySQL on the same server so I expect that I will get some performance improvement. That's a pretty good tip about stored procedures though...nice workaround perhaps. I'm wondering if the kind of locking that you use would have an impact on that (table locking vs. record locking).

mathmarker wrote:

NO. It's not. Apache running in pre-fork mode (the default setting with Apache 2 and the only safe way to run PHP) won't take advantage of the dual cores.

Yes it will.

Even in prefork mode, Apache STILL uses multiple threads. The only difference is, that in threaded mode, apache uses fewer processes - this is sometimes more memory efficient (e.g. especially with PHP and stuff).

The difference is in safety - PHP claims it's not necessarily thread safe and doesn't recommend the usage of threaded mode (which is pretty bonkers, as on win32, they do!).

You're better off with a 3.0 ghz single core than a 2.0 ghz dual core. Your PHP processes -- which account for most of the CPU activity -- can't take advantage of both cores.

I can't say whether one faster CPU is better than two slower ones - you MUST benchmark YOUR application to determine it.

Comparing performance with some of the cores disabled (on the same box) will enable like-for-like comparisons.

What if you're running MySQL on the same server?
Yes, you'll benefit from the dual cores. MySQL is mult-threaded and will utilize both cores without your changing a thing.

Irrelevant. MySQL IS multithreaded, but that's not relevant. Even if it weren't, multiple processes can still use multiple cores.

How can I speed my LAMP application on a multi-core system?
Upgrade to MySQL 5

Upgrading to mysql 5 will not necessarily increase performance. Once again, developers should benchmark their own application on 4 vs 5 if they're considering upgrading (NOTE: they should also do rigorous testing before such an upgrade- it carries significant risk).

and look for places to move PHP database recordset handling or logic into MySQL stored procedures.

AGain, that won't necessarily help.

Note, you don't have to do any fancy multi-threading coding. Since MySQL itself is multi-threaded it automatically can make use of both cores anytime its dealing with concurrent query requests from multiple clients.

Except if you have lock contention in mysql, which is a whole other topic.

None of what mathmarker said is necessarily true, and some of it is necessarily false. Please check these things carefully in future.

Don't take peoples word for it (especially mine) - remember - benchmark YOUR application, don't rely on someone else's benchmarks of their application or synthetic ones.

Mark