Monday morning I got a number of frantic calls about a CentOS LAMP server that was down. When I tried to ssh into the server using puTTY, the server would prompt me for username and password but rejected valid credentials. I called the hosting company and asked them for a manual reboot which appears to have the server functioning normally again. I logged in as root and checked to see if any commands were in the bash history and nothing appears to be in there. The sessions table (the reason for previous crashes) was fine.
I'm wondering two things and hope the community might offer some advice:
1) Is there some way to have a remote machine continually check this server and notify me when it stops serving pages properly? I'm thinking a ping is not enough. I want to be sure that php, mysql, mail, etc. (everything) are working properly. Is this a common service that people acquire? Is it configurable to test various services? Is it secure?
2) What should I be looking at to determine if the machine was compromised? I realize that a shrewd hacker would clear any bash history and otherwise cover his tracks.
3) Is there some reliable procedure or heuristic process I can use to determine the reason for this crash?
Any thoughts would be greatly appreciated.
