Server Crashing - trying to locate guilty items

sneakyimp

I'm trying to determine why a server is crashing every night. Looking at the output of top right around crash time (which, unfortunately, is never consistent), I see load averages from 3 to 6. What's puzzling me is the presence of some items in the process list which appear to be chewing resources.

In particular, the 'gzip' process (with no visible args) appears to be devouring enormous CPU resources. There's also a gtar process that's taking a very very long time.

top - 02:00:31 up 15:42,  1 user,  load average: 2.99, 2.26, 2.48
Tasks: 127 total,   3 running, 124 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.0%us,  1.8%sy,  2.5%ni, 23.3%id, 62.2%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   4046580k total,  4021396k used,    25184k free,    18272k buffers
Swap:  2104504k total,      120k used,  2104384k free,  3085548k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28963 mysql     15   0  635m 352m 4452 S  6.3  8.9  21:17.16 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.opreum.com.pid --skip-external-locking
 8902 root      34  19  4156  728  352 R  5.0  0.0  11:41.62 gzip
  502 root      10  -5     0    0    0 D  0.3  0.0   0:38.76 [kjournald]
  258 root      10  -5     0    0    0 S  0.3  0.0   0:20.30 [kswapd0]
23733 root      34  18 51436 9952 1636 S  0.0  0.2   0:15.82 /scripts/cpbackup
 4571 nobody    18   0  417m  13m 6416 S  0.3  0.4   0:11.59 /usr/local/apache/bin/httpd -k start -DSSL
 4574 nobody    18   0  417m  14m 6416 S  0.0  0.4   0:11.56 /usr/local/apache/bin/httpd -k start -DSSL
 4578 nobody    22   0  417m  14m 6420 S  0.7  0.4   0:11.56 /usr/local/apache/bin/httpd -k start -DSSL
 4576 nobody    18   0  353m  14m 6420 S  0.3  0.4   0:11.53 /usr/local/apache/bin/httpd -k start -DSSL
 4579 nobody    23   0  417m  14m 6416 S  0.0  0.4   0:11.51 /usr/local/apache/bin/httpd -k start -DSSL
 8901 root      34  19 21024 1020  840 R  0.0  0.0   0:09.38 /bin/gtar pczf siteuser.tar.gz siteuser
 2613 root      15   0     0    0    0 D  0.0  0.0   0:04.16 [pdflush]
 4394 root      18   0  194m 133m 6424 S  0.0  3.4   0:03.40 /usr/sbin/clamd
 3237 named     21   0  161m 4672 1952 S  0.0  0.1   0:02.71 /usr/sbin/named -u named

What is up with the gzip?

The process /bin/gtar pczf siteuser.tar.gz siteuser appears to be running as root but I don't see it in the root-level crontab. Perhaps its forked by some backup script? How do I locate it the source of this process?

In this snapshot, notice the transient php scripts such as "/usr/bin/php /home/siteuser/public_html/client/vehicle_detail.php":

root@server [/home/siteuser]# top
top - 02:19:54 up 16:02,  1 user,  load average: 3.06, 2.86, 2.84
Mem:   4046580k total,  4017556k used,    29024k free,    15672k buffers
Swap:  2104504k total,      104k used,  2104400k free,  3143504k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 8902 root      35  19  4156  728  352 R 39.8  0.0  25:18.24 gzip
14449 siteuser   17   0  132m  12m 7012 D  3.0  0.3   0:00.03 /usr/bin/php /home/siteuser/public_html/client/vehicle_detail.php
14450 siteuser   17   0     0    0    0 Z  3.0  0.0   0:00.03 [code=php] <defunct>
14451 siteuser   17   0     0    0    0 Z  3.0  0.0   0:00.03 [code=php] <defunct>
  502 root      10  -5     0    0    0 D  1.0  0.0   0:44.75 [kjournald]
 4574 nobody    18   0  419m  16m 6416 S  1.0  0.4   0:14.65 /usr/local/apache/bin/httpd -k start -DSSL
28963 mysql     15   0  636m 353m 4452 S  1.0  9.0  22:00.43 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.opreum.com.pid --skip-external-locking

I've always been under the impression that PHP runs as a module within the apache process. Is there some configuration that would cause PHP to run as its own process?

DaveRandom

If any of those processes were being run by cron, there would be a correponding crond forked process controlling it.

I'm guessing gzip is being run without args because gtar has called it and is writing/reading the data via STDIN/STDOUT.

You can invoke the php binary directly from command line and it will show as a 'php' process rather than httpd. I believe there are some configurations of CGI mode that will also produce this.

What is '/scripts/cpbackup' doing (from the output of top)? Also, clamd seems to be using a lot more memory than it normally does in my experience...

sneakyimp

Thanks so much for your response. Very helpful.

DaveRandom;10973362 wrote:
If any of those processes were being run by cron, there would be a correponding crond forked process controlling it.

Can you elaborate? Do you mean to say that I'd see a line in the top output that says crond and somehow corresponds? I think the critical task here is locating the source of these processes to learn more about them and possibly disable or alter them.

DaveRandom;10973362 wrote:
I'm guessing gzip is being run without args because gtar has called it and is writing/reading the data via STDIN/STDOUT.

I think you may be right -- thanks for that. I have ruled out that the gzip process might be running in conjunction with apache which was my prior suspicion. Someone added RAM and rebooted the server and the gzip task is no longer present.

DaveRandom;10973362 wrote:
You can invoke the php binary directly from command line and it will show as a 'php' process rather than httpd. I believe there are some configurations of CGI mode that will also produce this.

I'm aware of the command line invocation, but I don't think anyone is sitting around typing these buggers in -- they are constantly appearing and disappearing. Can you elaborate on the CGI mode setup? Any additional detail would be much appreciated.

DaveRandom;10973362 wrote:
What is '/scripts/cpbackup' doing (from the output of top)? Also, clamd seems to be using a lot more memory than it normally does in my experience...

I'm not entirely sure what all that script is doing. I do know that it is either the webhost manager or cpanel backup functionality. It's quite elaborate and is probably responsible for at least some of the gtar/gzip functionality. Unfortunately the hosting company in this case appears to be kind of clueless about how to configure this sort of thing. I've proposed that we remove all but the most crucial backup processes to give the server's hard drive a break and then maybe install a new hard drive and route backups there.

sneakyimp

OH...also note that the transient php scripts have owner 'siteuser' rather than 'nobody' (which is what apache runs as. There might be something like suphp in place or something? Really weird.

DaveRandom

sneakyimp wrote:
Can you elaborate on the CGI mode setup? Any additional detail would be much appreciated.

If it is a WHM-based, cgi-mode PHP is probably irrelevant, but TBH it's not my bag anyway - I always run PHP as an Apache module, and can't offer any useful info on running as a CGI above what the docs say. I only know that I have seen, on a server that does use cgi-mode, all my scripst being invoked as separate PHP processes - which was disasterously inefficient. The WHM developers certainly know better than that, I feel...

sneakyimp wrote:
Do you mean to say that I'd see a line in the top output that says crond and somehow corresponds?

I usually do. I will post the output of top from one of my boxes up here tomorrow (can't easily get access to it at the moment) so you can see what I mean.

I'll look into this more/play around with some of my less important machines and get back to you.

sneakyimp

Thanks again for your input. I'd love to see any examples you have of the cron-owned processes or the cgi-mode php scripts. If it is in fact disastrously inefficient, I'd like to see if I can change that configuration. Given that the server has been crashing, I'd like to free up any resources I can.

sneakyimp

If you have a moment, DaveRandom, I'd love to see the processes you were talking about.