This is what it looks like when you need RAM

[sneakyimp]

Just launched a site. Never seen a load average quite so high:

top - 22:32:02 up 40 min,  2 users,  load average: 286.56, 250.20, 147.30
Tasks: 457 total,   1 running, 456 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.6%sy,  0.0%ni,  0.0%id, 98.7%wa,  0.0%hi,  0.6%si,  0.0%st
Mem:   1927992k total,  1920336k used,     7656k free,     2592k buffers
Swap:  3863624k total,  3379528k used,   484096k free,    20740k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                           

 7843 root      16   0 13008 1224  692 R  0.7  0.1   0:04.40 top                                                

 9375 nobody    16   0  206m 9580 4644 D  0.3  0.5   0:01.59 httpd                                              

 9429 nobody    17   0  208m  12m 5348 D  0.3  0.7   0:02.50 httpd                                              

 9600 nobody    16   0  194m  11m 5232 D  0.3  0.6   0:01.38 httpd                                              

10023 nobody    16   0  207m 9.8m 4132 D  0.3  0.5   0:01.90 httpd                                              

10177 nobody    17   0  205m  11m 4572 D  0.3  0.6   0:01.18 httpd                                              

10193 nobody    16   0  190m 9.9m 4552 D  0.3  0.5   0:00.80 httpd                                              

10195 nobody    16   0  193m  14m 4864 D  0.3  0.8   0:01.19 httpd                                              

10280 nobody    16   0  191m 9188 4500 D  0.3  0.5   0:00.57 httpd                                              

10284 nobody    16   0  195m  17m 4588 D  0.3  1.0   0:00.83 httpd                                              

10485 nobody    16   0  205m  30m 4616 D  0.3  1.6   0:01.04 httpd                                              

10543 nobody    16   0  207m  31m 4596 D  0.3  1.7   0:00.64 httpd                                              

10564 nobody    18   0  205m  26m 3672 D  0.3  1.4   0:00.38 httpd                                              

10772 root      18   0 74824  460  308 D  0.3  0.0   0:00.07 crond                                              

10780 root      18   0 37452 1212  152 D  0.3  0.1   0:00.04 dcpumon                                            

    1 root      16   0 10352  568  520 D  0.0  0.0   0:00.87 init                                               

    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0                                        

    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0                                        

    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                         

    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/1                                        

    6 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1                                        

    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1                                         

    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/2                                        

    9 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/2                                        

   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2                                         

   11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/3                                        

   12 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/3                                        

   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3                                         

   14 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/4                                        

   15 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/4                                        

   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/4                                         

   17 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/5                                        

   18 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/5                                        

   19 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/5                                         

   20 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/0                                           

   21 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/1                                           

   22 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/2                                           

   23 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/3                                           

   24 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/4                                           

   25 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/5                                           

   26 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khelper                                            

   42 root      11  -5     0    0    0 S  0.0  0.0   0:00.08 kthread                                            

   51 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0                                          

   52 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/1                                          

   53 root      10  -5     0    0    0 S  0.0  0.0   0:00.06 kblockd/2                                          

   54 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/3                                          

   55 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/4                                          

   56 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/5                                          

   57 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid                                             

  168 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/0                                           

  169 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/1                                           

  170 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/2                                           

  171 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/3                                           

  172 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/4                                           

  173 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/5                                           

  176 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd                                              

  178 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod                                            

  289 root      15   0     0    0    0 S  0.0  0.0   0:00.00 khungtaskd                                         

  292 root      10  -5     0    0    0 D  0.0  0.0   1:09.81 kswapd0                                            

[dalecosp]

Hee hee, just put it on FreeBSD instead :-P

[sneakyimp]

dalecosp;10972910 wrote:

Hee hee, just put it on FreeBSD instead :-P

Whaddya mean?

[dalecosp]

Looks like it's hosted on Linux? FBSD probably would manage the load a tad better. Albeit, I've never seen top stats with a load avg that high either. Must be a big deal ... congrats :-)

[sneakyimp]

The launch was fairly big, but I think there might be some memory leaks causing the issue. The load average started out reasonably and slowly grew to this high number due to the paging of memory. The swap grew and grew and the number of processes waiting started to rise.

Adding RAM solved the problem for the most part. Still some weirdness.

[dalecosp]

Well, not such a debacle as what I just DOH'ed. Maybe, at least.

I think I just circular referenced two PHP scripts so that they were redirecting to each other continually. Not something foreseeable, exactly; I'm not the master of the server's configuration, having only started with this site last month. But at any rate, the CentOS box under it ramped CPU up until we had to get the support guys to reboot it remotely. And this is our "flagship world wide" site.

Needless to say, I switched strategies. Next up, to talk to the IT boss about setting up a devel box. 😉

[sneakyimp]

DOH! Do you mean to say that you were using header('location: other.php') ? Or was it a mod_rewrite thing?

Dev boxes rule. The domain or subdomain might differ, but it sure beats putting the dev site in a subdirectory and having path issues. I like to allocate one from the rackspace cloud for around $10 a month.

I also like to set up sites I'm working on here on my office dev box. My prior desktop (dual core machine, 2GB RAM, 2.66Ghz) has been repurposed as a development box and protools LE rig. I have 4 bootable OSes on it:
Win XP for development
Win XP for protools
Ubuntu 32-bit
Ubuntu 64-bit

I can just fire up the machine to one of the ubuntu boots and usually get a site copied via SSH and either rsync command or scp command in pretty short order.

Then, on my windows desktop I edit my HOSTS file so that it points 'devdomain.com' to the IP address on my LAN (e.g., 192.168.1.2).

I also have a Mac with Apache/PHP/MySQL manually installed.

My dev boxes may differ in configuration from the production box. Having an exact copy of your production server for your dev server is ideal.

[dalecosp]

sneakyimp;10973832 wrote:

DOH! Do you mean to say that you were using header('location: other.php') ? Or was it a mod_rewrite thing?

Dev boxes rule. The domain or subdomain might differ, but it sure beats putting the dev site in a subdirectory and having path issues. I like to allocate one from the rackspace cloud for around $10 a month.

I also like to set up sites I'm working on here on my office dev box. My prior desktop (dual core machine, 2GB RAM, 2.66Ghz) has been repurposed as a development box and protools LE rig. I have 4 bootable OSes on it:
Win XP for development
Win XP for protools
Ubuntu 32-bit
Ubuntu 64-bit

I can just fire up the machine to one of the ubuntu boots and usually get a site copied via SSH and either rsync command or scp command in pretty short order.

Then, on my windows desktop I edit my HOSTS file so that it points 'devdomain.com' to the IP address on my LAN (e.g., 192.168.1.2).

I also have a Mac with Apache/PHP/MySQL manually installed.

My dev boxes may differ in configuration from the production box. Having an exact copy of your production server for your dev server is ideal.

Oh, yeah ... when I was doing mostly my own work, I sat on the dev server in my office. Plenty of horses for development and I could still watch Youtube to my heart's content......

Not so fortunate at work. I'd take a picture, but you'd maybe not believe me, still. I mean, there's a desk/cubicle two people down from me, and this morning I spotted a FoxPro boxed set. These folks don't know Linux/BSD if it hits them in the face. But then, I suppose that's why I have the job.

[dalecosp]

Oh, and I think it was both mod_rewrite and header('location: .... one on purpose, and the other I'd forgotten about....