db logging class - is it unsafe to show output of uniqid? should i use text columns?

sneakyimp

Payment gateways and cloud APIs can be finicky sometimes and often return complex results depending on elaborate circumstances. In the past, when I've tried to charge someone's credit card and something goes wrong -- e.g., the card was declined or the POST operation to the gateway failed to validate the cert, etc. -- I have written whatever relevant information I can to a text log. Text files are a pain. You don't want them in the web root so you have to put them outside the web root which means you have to either download them via SFTP or you have to login via CLI to inspect them or you have to write a script to search them, etc.

A DB log seems like a really prudent idea here. Create a DB_Log object and use it to log any interesting events that may happen during a specific page request. At the end of the page request, if there's a failure, then give the user some watered-down apology that cannot be used by hackers to game your system and give them some unique id that they can reference if they really think there's something wrong and they want to follow up.

I'm thinking DB_Log will be instantiated at the beginning of a page request and will generate some unique ID that is safe for even hackers to see. Each time it encounters some log-worthy situation, it will write a db record with the info and the unique id. When the page request ends in some kind of caught exception, the DB_Log object can be inspected for the unique id so that you show it ot the user.

Something like this:

<?php
// the DB_Log constructor will create some page_request_id that is stored with all log messages from this page request
$log = new DB_Log("some-log-category");
try {

  // try fancy stuff in here
  if ($fancy_api_response->isError()) {
    $log->write("fancy_api_response returned an error: " . $fancy_api_response->getErrorDetails());
  }

  if ($all_hell_breaks_loose) {
    throw new Exception("OH DAMN IT'S THE END OF THE WORLD");
  }

} catch (Exception $e) {
  $log->write("An exception was caught: " . $e);

  die("Sorry, dear user, but your attempt failed. If you want to follow up, contact tech support and use reference number " . $log->page_request_id);
}
?>

Any thoughts on this approach are welcome, but I have two big questions:
would it be safe to just use [man]uniqid[/man] for the page_request_id or is it a bad idea to show the output of this function? Seems like it might help hackers to game my server if they knew its values.
if uniqid is not safe, can anyone suggest a good approach to generating page request IDs that are short, unique, and easily communicated over the phone?
* I imagine this table might grow quite large and wonder about the reliability and efficiency of using text columns. I'm pretty sure 255 chars is going to be too short for some errors. Is it advisable to use a text field in this context? Can anyone foresee any risks?

ixalmida

If you are creating an arbitrary ID that only relates to the error log, I don't see how security would be a problem. For that matter, an auto-increment ID should work just fine. If a hacker gets access to the database and can read the log, you've got bigger problems. Assuming there is no place for an end user to read the log using the ID, there should be no worry about anyone reading the log. Hopefully, you have smart enough tech support people that they will know a legitimate tech support call from a fake one.

If you are getting so many errors that the log fills up fast, again...you have bigger problems. Regardless, I would just use a text field and purge your log data regularly (in a monthly cron job, for example).

I don't quite follow on not being able to read a file outside the web root. If you can write them there, you can read them there as well.

sneakyimp

Thanks for your response.

ixalmida;11050077 wrote:
If you are creating an arbitrary ID that only relates to the error log, I don't see how security would be a problem. For that matter, an auto-increment ID should work just fine.

I'm not sure I agree. [man]uniqid[/man] is a basic PHP function that can be reasonably relied upon to always generate an id that has not been seen before. The problem is that it is apparently time-based and may reveal insights into the entropy conditions on one's server -- or at least that's my concern. Most encryption relies on the generation of random numbers via those entropy conditions and if a savvy hacker were to gain insight into these conditions, it might assist them in cracking other more important encryption -- like the encryption that protects financial transactions.

As for just using an auto-increment ID, I believe that is patently unsafe because the predictable sequence of numbers would make it easy for a malicious person to predict the next ID and possibly gain access to some such report through social engineering. If you can predict the ID on some trouble ticket, a gullible employee is more likely to mistake knowledge of that ID for account ownership. They might then reveal some other sensitive information.

ixalmida;11050077 wrote:
If a hacker gets access to the database and can read the log, you've got bigger problems. Assuming there is no place for an end user to read the log using the ID, there should be no worry about anyone reading the log. Hopefully, you have smart enough tech support people that they will know a legitimate tech support call from a fake one.

Agreed that access to the log presents bigger problems -- and this is not likely. Sadly, I have no control over the employees who might have access to this error log on the administrative side. There will be a way to initiate a customer support ticket for an ID, and hackers might be able to exploit that if the log ids are predictable.

ixalmida;11050077 wrote:
If you are getting so many errors that the log fills up fast, again...you have bigger problems. Regardless, I would just use a text field and purge your log data regularly (in a monthly cron job, for example).

No idea yet how many log entries will be written, and I have in fact taken the cron-purge approach in the past. Mostly worried about scalability here. What happens if we have 5 million users, each with 10 log entries a day? Can such a system handle 50 million entries daily (1.5B/mo) ?

ixalmida;11050077 wrote:
I don't quite follow on not being able to read a file outside the web root. If you can write them there, you can read them there as well.

I was mostly complaining about the inconvenience of text files for logging and trying to make the point that if the log files are outside the web root, you'd have to concoct some script to search them or some means of download. Basically, text files for logging are pretty inconvenient.

Bonesnap

I would use text columns. The alternative is to administer a rule that your log entries are at most X length. That means either truncating your messages or removing info from them to make them fit should they exceed that size. I think at that point you run the risk of making your log entries useless or at the very least not as helpful. I imagine you'd want to include as much information in these entries as possible to assist both tech support and by extension the user. Also I would think that it would be difficult to decide what to remove from the entry at the code level.

Regarding your scenario of 5 million users with 10 entries per day, which part of the scalability are you concerned about? Time required insert the record, retrieving it, etc.? To put things into perspective, when I was playing around with my hash database the most I got up to was about 800 million rows. Inserts were taking their time, that's for sure, but that was because I was doing an INSERT IGNORE (to avoid duplicates) and I was inserting 10s of millions of rows at a time per transaction. Reading the database, as in, retrieving a plain-text password based on a hash, pretty much was never affected - I could still get a corresponding plain-text password in a fraction of a second. And this was just on my personal computer - I imagine you have your application on hardware that's much more powerful.

ixalmida

Ah, I understand better why you want to use uniqid. Have you considered openssl_random_pseudo_bytes()? It is cryptographically strong with a reasonable expectation of uniqueness (depending on how many characters it returns). Here's a good post on that.

I rarely consider scalability to be an issue these days. If it is too big to host on your hardware, move it to the cloud. The company I work for logs hundreds of millions of clicks per month (before any data transformation, reporting, etc.). If our website performance ever dips, AWS just spins up more instances and the load gets distributed evenly.

EDIT: By the same token, AWS spins the instances down again when the load isn't as heavy, so you don't keep paying for power you don't need.

sneakyimp

ixalmida;11050091 wrote:
Ah, I understand better why you want to use uniqid. Have you considered openssl_random_pseudo_bytes()? It is cryptographically strong with a reasonable expectation of uniqueness (depending on how many characters it returns). Here's a good post on that.

See that's just the problem: I don't want to expose my server's random byte generating functions too directly to evil people who might be able to analyze it to find vulnerabilities -- i.e., ways that it's not so random so they can attack my encrypted communications. I'm not entirely sure how crypto attacks work, but I have a vague understanding that if they can predict your random numbers, they can crack your encryption.

ixalmida;11050091 wrote:
I rarely consider scalability to be an issue these days. If it is too big to host on your hardware, move it to the cloud. The company I work for logs hundreds of millions of clicks per month (before any data transformation, reporting, etc.). If our website performance ever dips, AWS just spins up more instances and the load gets distributed evenly.

In my opinion, scalability is always a consideration. The 'spinning up new instances' concept assumes that you've constructed your web app such that you don't have to worry about race conditions or DB bottlenecks. If your content is read-only this is easy to do -- you just replicate the content. If your site works with user input, there are issues of data consistency to wrangle with.

ixalmida;11050091 wrote:
EDIT: By the same token, AWS spins the instances down again when the load isn't as heavy, so you don't keep paying for power you don't need.

Sounds like you are also using a load balancer to me.

Weedpacket

I note that uniqid does allow specifying an additional entropy source.

sneakyimp wrote:
* I imagine this table might grow quite large and wonder about the reliability and efficiency of using text columns. I'm pretty sure 255 chars is going to be too short for some errors. Is it advisable to use a text field in this context? Can anyone foresee any risks?

Depends on how the DBMS stores variable-length fields; MySQL InnoDB's "Barracuda" store variable-length fields separately from the rest of the table and has the option of compressing them. PostgreSQL's TOAST also does this, and also has the option of storing the data within the main table if it is short enough; it's also worth noting that PostgreSQL's character types are all stored the same way regardless of their semantics.

If you've got a consistent set of error messages that can be identified by an ID number, then you can store that (possibly with additional columns for variable parts of the error string) along with a lookup table that matches ID to error message text. This would make searching for particular classes of error easier.

sneakyimp

Thanks for your response, Weedpacket.

Weedpacket;11050095 wrote:
I note that uniqid does allow specifying an additional entropy source.

I don't think [man]uniqid[/man] lets you specify your entropy source but rather lets you specify TRUE for the second param if you want more entropy form whatever source it is derived from. My question really is whether it is poor security practice to reveal the output of uniqid to visitors? Seems to me it might give shrewd hackers more insight into the entropy generation methods of your server.

Also, asking for more entropy from uniqid results in a 23-char output instead of a 13-char output, which would be more difficult to communicate over the phone. It is my hope to generate the shortest, most easily readable random unique ID that I can. This unique ID will be used for all log entries entered in connection with a single page request. It will be stored in user_friendly_id in this table:

CREATE TABLE `db_log` (
  `id` int(15) unsigned NOT NULL AUTO_INCREMENT,
  `creation_microtime` decimal(16,5) unsigned NOT NULL,
  `user_friendly_id` varchar(13) NOT NULL,
  `severity` enum('INFO','WARNING','DANGER') NOT NULL,
  `log_entry_type` varchar(20) NOT NULL,
  `log_entry` text NOT NULL,
  `related_db_table` varchar(45) DEFAULT NULL,
  `related_db_id` int(15) unsigned DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `log_entry_type` (`log_entry_type`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8;

log_entry_type is another free-form field that can be used to categorize log entries further. E.g., they might be of type PAYMENT or perhaps CDN_API problems or FILE_UPLOAD problems.

Weedpacket;11050095 wrote:
Depends on how the DBMS stores variable-length fields; MySQL InnoDB's "Barracuda" store variable-length fields separately from the rest of the table and has the option of compressing them. PostgreSQL's TOAST also does this, and also has the option of storing the data within the main table if it is short enough; it's also worth noting that PostgreSQL's character types are all stored the same way regardless of their semantics.

Helpful info, although I'm wondering how many records I might expect to have in my db_log table before it starts to slow things down if the db is hosted form a standard hard drive. A single page request might want to enter 3-5 entries if it runs into some trouble that is log-worthy.

Weedpacket;11050095 wrote:
If you've got a consistent set of error messages that can be identified by an ID number, then you can store that (possibly with additional columns for variable parts of the error string) along with a lookup table that matches ID to error message text. This would make searching for particular classes of error easier.

My two classification columns in this table are user_friendly_id, which will identify all errors that might be encountered with a single page request, and then log_entry_type, which can be used for whatever other classification scheme we devs might want.

sneakyimp

I'm also wondering if there is any way for a function to determine the file and line from which it was called. I've looked at [man]debug_backtrace[/man] and it doesn't provide file or line number. It's kind of a nuisance to always have to send FILE and LINE as parameters to my $dblog->write() function.

Derokorian

sneakyimp;11050111 wrote:
I'm also wondering if there is any way for a function to determine the file and line from which it was called. I've looked at [man]debug_backtrace[/man] and it doesn't provide file or line number. It's kind of a nuisance to always have to send FILE and LINE as parameters to my $dblog->write() function.

Um, you might want to look again. The example output is:

array(2) {
[0]=>
array(4) {
    ["file"] => string(10) "/tmp/a.php"
    ["line"] => int(10)
    ["function"] => string(6) "a_test"
    ["args"]=>
    array(1) {
      [0] => &string(6) "friend"
    }
}
[1]=>
array(4) {
    ["file"] => string(10) "/tmp/b.php"
    ["line"] => int(2)
    ["args"] =>
    array(1) {
      [0] => string(10) "/tmp/a.php"
    }
    ["function"] => string(12) "include_once"
  }
}

I see file and line right there...

sneakyimp

Hm. It doesn't appear in my Object::method which is where I'll be calling it from probably:

array(3) {
  [0]=>
  array(5) {
    ["function"]=>
    string(3) "foo"
    ["class"]=>
    string(4) "Home"
    ["object"]=>
    &string(16) "object blah blah"
    ["type"]=>
    string(2) "->"
    ["args"]=>
    array(0) {
    }
  }
  [1]=>
  array(4) {
    ["file"]=>
    string(44) "/var/www/my-site/system/core/CodeIgniter.php"
    ["line"]=>
    int(508)
    ["function"]=>
    string(20) "call_user_func_array"
    ["args"]=>
    array(2) {
      [0]=>
      &array(2) {
        [0]=>
        &string(16) "object blah blah"
        [1]=>
        string(3) "foo"
      }
      [1]=>
      &array(0) {
      }
    }
  }
  [2]=>
  array(4) {
    ["file"]=>
    string(31) "/var/www/my-site/html/index.php"
    ["line"]=>
    int(291)
    ["args"]=>
    array(1) {
      [0]=>
      string(44) "/var/www/my-site/system/core/CodeIgniter.php"
    }
    ["function"]=>
    string(12) "require_once"
  }
}

EDIT: the file in this case would be
/var/www/my-site/application/controllers/admin/Home.php
and the line would (probably) be:
21

sneakyimp

So I'm using debug_backtrace, which may or may not provide the file/line from which a function is called. Still not sure when it does and when it does not provide this information. I must take care to check if the value is supplied:

		// find out from whence this function was called
		$bt = debug_backtrace();

	// the file
	$file = isset($bt[0]['file']) ? $bt[0]['file'] : NULL;
	// take note of the line
	$line = isset($bt[0]['line']) ? $bt[0]['line'] : NULL;

Still wondering: Would showing a hacker the output of uniqid give him/her any insight to help crack my server's encrypted communications?

Perhaps this is a philosophical question.

dalecosp

sneakyimp;11050555 wrote:
Still wondering: Would showing a hacker the output of uniqid give him/her any insight to help crack my server's encrypted communications?

Perhaps this is a philosophical question.

PHP Manual wrote:
Gets a prefixed unique identifier based on the current time in microseconds.

Warning
This function does not create random nor unpredictable strings. This function must not be used for security purposes. Use a cryptographically secure random function/generator and cryptographically secure hash functions to create unpredictable secure IDs.

Caution
This function does not generate cryptographically secure tokens, in fact without being passed any additional parameters the return value is little different from microtime(). If you need to generate cryptographically secure tokens use openssl_random_pseudo_bytes().

So, unless you have a cracker (hackers are, after all, someone who makes furniture with an axe) who can determine the contents of private key stashes via a UNIX Timestamp, I'd have to answer your question with a fairly resounding "no", especially if you're not calling uniqid() with the more_entropy flag set. 🙂

sneakyimp

dalecosp;11050575 wrote:
So, unless you have a cracker (hackers are, after all, someone who makes furniture with an axe) who can determine the contents of private key stashes via a UNIX Timestamp, I'd have to answer your question with a fairly resounding "no", especially if you're not calling uniqid() with the more_entropy flag set. 🙂

I had seen that. To say that uniqid does not provide cryptographically secure strings does NOT automatically prove that it does not reveal sensitive information about the state of one's computer. I'm not sure I agree resounding at all.

dalecosp

What's so dangerous about a timestamp? Are you tagging sensitive information with it?

sneakyimp

dalecosp;11050579 wrote:
What's so dangerous about a timestamp? Are you tagging sensitive information with it?

I won't pretend I know a lot about it, but I have heard that crypto attacks can be assisted if crackers have extra information about your system. See side-channel attack.

I am contemplating using the output of uniqid as an alternate unique key that I can show to users as a 'ticket' if they want to follow up on problems associated with a page request. I'll write generate this ticket number and use it to write any log entries that may be necessary in conjunction with some user action. If a user presents me with their ticket number I can look up all the matching entries in my db log. In theory this would be safe because the user would not be privy to sensitive information (e.g., a payment gateway's response to a credit card transaction attempt) but my employees could use this ticket number to help them with their problem. Am I tagging sensitive information? Yes I suppose -- I hope my explanation answers that question.

Am I using uniqid to encrypt anything? NO. In this context it would merely be an alternate to the record's id. I would prefer something random, but am essentially concerned about revealing too state information about my server. It might be very easy for an attack to generate as many of these ticket ids as they want simply by repeatedly generating some error.

I'm certainly open to alternatives. Ideal properties of ticket ids:
always unique
non-sequential
short, easy to read over the phone
easily generated without consuming a lot of system resources.

EDIT: Having read that article on side-chain attacks, it would seem that knowing an accurate time stamp on the server is not especially risky. It's the duration of certain calculations that presents risk. Still, as I am no expert here, I wonder if revealing the output of uniqid might not be a good idea.