DON'T DO THIS! (include of death)

mnuzum

Well, here's a really silly mistake.

For 48 hours I thought my server was under DOS attack [Denial Of Service]. In a way it was, but it was our own server attacking itself.

My server was configured for 300 max child processes, which is a little too high. Saturday PM, All activity on the server suddenly came to a screeching halt. I noticed that the server was out of memmory and there were 300 apache processes running. I figured it was a runaway glitch and rebooted.

Later on it happened again. I rebooted the server again and changed my passwords just in case it was someone trying to hack in. I checked the processes, cron, at and the command history of my admin users. Nothing in the slightest to indicate a hack, or a hack attempt.

This morning, I went to the server and tried to view a web page, sure enough, it took 5 minutes before the first page showed up. Instead of rebooting the server, I checked the running processes by doing a 'top -bn 1' and saved the output to a file.

I then did /etc/rc.d/init/http restart to kill all the apache children and restart. The server instantly snapped back into shape. I did my 'top -bn 1' and emailed the before and after output to myself to compare the differences.

Within 10 minutes however, my server was fully loaded with 300 httpd processes. What could this be? Because I host several Christian related websites, I suddenly started thinking this had something to do with the Air raides in the middle east :-) I went to several of my favorite christian websites, the first of which was Gospelcom.net. It just so happend that gospelcom was doing routine maintenance (which I did not know at the time) and there web site wasn't working.

Of course I thought the worst.

I went into my httpd.conf file and changed the max request to 50 so that the server would operate its other services like normal. I then began examining the logs. I was trying to call people I knew to find out if their websites were working. I soon figured out it was just my site.

What could it be? Well, finally, I took the time to CAREFULLY look at the access files. I found something happening that was a little mysterious:
YYY.followers.net 64.77.36.XXX - - [08/Oct/2001:08:18:44 -0700] "GET /web/adodb/adodb.inc.php HTTP/1.0" 200 - "-" "PHP/4.0.6"

It just so happens that 64.77.36.XXX is my own IP address. So something on my own server was attacking the server. I thought it was funny that they were trying to get this adodb.inc file again and again. It didn't have anything important, and if they wanted it that bad, they shoud just go to http://php.weblogs.com and get it themselves.

Then I looked at the path. Hmmmm... That's almost the same path I would use if I were trying to include the file in a PHP script.

I looked in the file that was uploaded last before the "attacks" came on Saturday. Sure enough, the line that should have read:
require("/web/adodb/adodb.inc.php");
instead read:
require("http://YYY.followers.net/web/adodb/adodb.inc.php");

Simple little mistake. 48 hours & 2-3 gallons of sweat later, it's kind of funny.

I'm not sure why this happend. Apparently the only way to stop it is to kill apache. It must have just happened that the developer who uploaded the file was checking the application occasionally to see if the server was working yet. Each time the script was run, it would start another round of "attacks" on the server.

Well, I hope that this experience saves you some time. Good luck.

Matt Nuzum

Anon

OK - so those of us who don't understand what you just explained (I don't do apache yet...): basically, you're saying that if we include a file, we should use a relative path and not a specific path? and this is because the server then has to contact itself via http? Maybe you could reduce it down to a real simple : don't do this: blah... (I'm stupid sometimes, forgive me!)

Anon

You might also want to check that files aren't getting included multiple times - use require_once and include_once.

Anon

no, what he is saying is he is a fuckup programmer....

do include files that include files that include the orginal file.

basically he had an infinate loop, fucked up the code and blamed it on bin laded just like everyone else does....

mnuzum

Well, Not exactly.

The programmer that did this is one of the most brilliant programmers I know. It was an honest mistake.

What I was trying to point out is that there is a serious problem with including files using the 'http' method if the files don't exist.

Sorry it took so long to explain, but I really had to give the whole story for my own benefit.

mnuzum

require("http://path/to/file")

was used instead of

require("/path/to/file")

Note the http://.

It caused some kind of wicked loop that kept making repeated requests again and again until there were no more avaiable server processes.

This caused all traffic to the webserver to stop.

Sorry for all the technical mumbo jumbo. I had to give all the details for my own benefit.

Matt Nuzum

[deleted]

Thanks for the warning!

It's always good to hear how someone else got into trouble (an how he got back out), so we can avoid doing the same.

Anon

he just told you NOT to use include("http:// or require("http:// and that is WRONG! these are perfectly fine to use and wont crash your server.

his implementation just created an infinate loop of includes.

this would be like people telling you to not use while() loops because one time they wrote one that had an infinate loop and took down the server.

this whole thread is silly.

mnuzum

For the second time in this thread, you've read more into my messages than I have stated.

I didn't tell anyone NOT to use this type of include.

I simply highlighted the difference between the local and remote versions.

If you use the remote include, a person should be careful to do it in a way that won't cause infinite loops.

This was a very simple mistake, and simple mistakes have a tendancy to be repeated by others.

The goal of the original post was to be here, in the message board, in case someone else in the future experiences a similar problem.

I appreciate the vast wealth of knowledge stored here and have benefitted from it numerous times. I want nothing more than to return the favor to some other person out there.

Matt Nuzum

Anon

thanks for the more simple explanation. Complements also on your tact and politeness with Mr. Kristopeit.

[deleted]

Speaking of tact and politeness, I quote M.Kristopeit in his first reply:

"no, what he is saying is he is a fuckup programmer.... "

I'd be quite pssed too if somebody said I was a fuckup programmer.

Anon

this isn't the place to post your coding problems, and how you fixed them, if the original problem was a mis-use of php.

the situation was this:
1) ahhhhhhhh my server is fucking up, terrorists did it.
2) tell everyone i'm having a problem, but don't post code or further explain the problem and what occured before it happened
3) realize the code had an infinate loop and tell everyone.

who could possibly benefit from this?

Anon

Matt,

Thanks for sharing your experience, I'll keep it in mind if any of my webservers get bogged down. And I'll be on the lookout for typos causing infinate loops. Maybe I can benefit from your experience somehow.

Later!