You guys are totally right...I should have been more specific in my question. Let me rephrase it:
Given that a faster server is probably cheaper than me totally re-examining my code (extra cost for upgrade to dual-core AMD 3800+ is about $720 per year), can I be sure of a performance gain for Apache/PHP/MySQL when I move to dual-core system? Basically, does a LAMP system really take advantage of a multi core/multi-proc system or will apache and mysql (the CPU-munching procs on my machine) still suffer from sequential execution?
More info: (also check here)
My server is a 2GHZ Celeron processor with 768 MB running Fedora Linux. The kernel is a bit old because it's been running for awhile (463 days since reboot!):
Linux myplan.myplan.com 2.4.22-1.2115.nptl #1 Wed Oct 29 15:42:51 EST 2003 i686 i686 i386 GNU/Linux
.
RAM
I know my server needs a little more RAM. The free command during peak hours indicates that there's only about 20MB of RAM available and about 120 MB of disk swap are in use. However, I don't get the feeling that adding a a few GB of RAM is really going to provide the performance capability to address our traffic growth moving forward.
DISK
As far as I can tell, the system is NOT I/O bound. The top command reveals the CPU usage rocketing to as high as 97% in short bursts. I have never seen the 'iowait' value in the top output be anything but 0.0%. Given that most queries to the db return only small amounts of data, I'm pretty sure that the disk drive speed is not at issue.
DB
As I just said, nearly all queries return only small amounts of information. I spent a lot of time working on my slower queries to try and optimize the data structures. Most of the queries that are not mine are from PHPBB. I'm CERTAIN some could be improved, but am wondering if it's worth my time and effort if I can get a significant boost from a $700 investment in server hardware.
BANDWIDTH
Hm. Hadn't yet considered that. The site is relatively image-intensive in its layout. Any quick tips on how to gauge that quickly?
CPU
As described above, the CPU frequently bursts into fairly high percentage ranges even during low to moderate traffic. The load average is typically 0.65 or higher during moderate times and I haven't yet been able to log on during peak usage to find how it's working then.
INEFFICIENT PROGRAMMING
HA! FAT CHANCE. But seriously I'm sure there could be performance improvements. For instance, I include a lot of functions in my init.php file for convenience that are not used on every page. I tend to be pretty attentive to inefficient situations though and preliminary performance profiling indicated that db queries and a few other select routines were orders of magnitude slower than the convenience shortcuts I took.
My admittedly cursory analysis says that, aside from poor programming or DB construction, RAM and CPU are the culprits. At the very least it would seem wise to test the very affordable option of a faster machine before totally reworking my code.
If there's some kind of awesome tool that could point out my bottleneck here, please let me know what it is!