UTF-8 and Me: Some questions

[schwim]

Hi there everyone!

In a "better late than never but really wished it had been earlier" scenario, I find myself validating a website after the fact. The site is a social sharing site and can be found here: http://schw.im . During my efforts, I was warned about lack of encoding. During my subsequent reading, I found that it was suggested that the char enc be UTF-8, if at all possible. I set that on the page and began getting errors concerning non-UTF-8 characters, making the page unable to be validated.

The site allows user submission of content and often, they copy-paste descriptions into the system. That's where I'm finding non-UTF-8 characters. apostrophes, quotes, foreign characters, things of this nature.

So I began looking up a solution via PHP to clean this up during content submission, which led me to htmlspecialchars . It looks to be what I need, but I'm not sure how to use it specifically for my needs. I already use it on the query to display the content with ENT_QUOTES to keep from breaking textareas and inputs, but It looks to me like I need to move it to the submission side and add a little bit to it to convert the non-UTF8 characters for safe display.

So my question is twofold. First, how would I structure the insert to include the conversion:

$custom_embed = mysqli_real_escape_string($link, htmlspecialchars(stripslashes($row_url['custom_embed']), ENT_QUOTES));

Secondly, Would this convert a non-conformant apostrophe to one that would display in UTF-8, or does it just throw a ? in there?

Thanks for your time!

[NogDog]

I copied this function from somewhere on the web to convert some of those M$ Word-specific characters, which might help:

<?php
function filterText($text)
{
   $search = array (
      '&',
      '<',
      '>',
      '"',
      chr(212),
      chr(213),
      chr(210),
      chr(211),
      chr(209),
      chr(208),
      chr(201),
      chr(145),
      chr(146),
      chr(147),
      chr(148),
      chr(151),
      chr(150),
      chr(133)
   );
   $replace = array (
      '&amp;',
      '&lt;',
      '&gt;',
      '&quot;',
      '&#8216;',
      '&#8217;',
      '&#8220;',
      '&#8221;',
      '&#8211;',
      '&#8212;',
      '&#8230;',
      '&#8216;',
      '&#8217;',
      '&#8220;',
      '&#8221;',
      '&#8211;',
      '&#8212;',
      '&#8230;'
   );
   return str_replace($search, $replace, $text);
}

Also, you'll likely want/need to let your database connection know you're talking UTF-8 now. Here's some info I collected a few years ago: http://www.ebookworm.us/utf8-in-php-and-mysql/

[Weedpacket]

There's also the [man]iconv[/man] function, which can be used to transliterate between character sets.

[schwim]

NogDog;11045453 wrote:

I copied this function from somewhere on the web to convert some of those M$ Word-specific characters, which might help:

<?php
function filterText($text)
{
   $search = array (
      '&',
      '<',
      '>',
      '"',
      chr(212),
      chr(213),
      chr(210),
      chr(211),
      chr(209),
      chr(208),
      chr(201),
      chr(145),
      chr(146),
      chr(147),
      chr(148),
      chr(151),
      chr(150),
      chr(133)
   );
   $replace = array (
      '&',
      '<',
      '>',
      '"',
      '&#8216;',
      '&#8217;',
      '&#8220;',
      '&#8221;',
      '&#8211;',
      '&#8212;',
      '&#8230;',
      '&#8216;',
      '&#8217;',
      '&#8220;',
      '&#8221;',
      '&#8211;',
      '&#8212;',
      '&#8230;'
   );
   return str_replace($search, $replace, $text);
}

Also, you'll likely want/need to let your database connection know you're talking UTF-8 now. Here's some info I collected a few years ago: http://www.ebookworm.us/utf8-in-php-and-mysql/

Hi there Nog and thanks very much for the reply.

After reading your page(thanks so much for that), I dumped my db, search & replaced latin1 and imported it. Everything seems to have worked properly in that regard.

I now need to deal with my input side. Do you think that I should both use your included function to convert the characters and then use htmlspecialchars / htmlentities to convert before db insertion? I've read the manual and it states that

htmlentities is identical to htmlspecialchars() in all ways, except with htmlentities(), all characters which have HTML character entity equivalents are translated into these entities.

Which means, I don't know which one to use 🙂

[NogDog]

I prefer to only use htmlentities() or htmlspecialchars() when actually outputting to HTML. If you do it to the data to be inserted into your database, you now bloat that data with HTML character entities that only mean anything to HTML, but not to any other use of your data. If all your data is UTF-8 and you are outputting HTTP headers indicating it's UTF-8, htmlspecialchars() is probably all you need.

[schwim]

Thanks, NogDog. Would you also suggest using the function you shared on the display end or does that stop illegal characters from being stored in the database, requiring me to use it on the insert end?

Also, reading the function page, it seems like UTF8 is the default now, but that it's good practice to define it. Does this fix me up for both setting charset as well as converting the quotes, so I don't break my form elements:

htmlspecialchars($myVariable, ENT_QUOTES,'UTF-8');

Thanks!

[NogDog]

Actually, it looks like the forum messed up that function definition by converting a lot of codes to actual characters. Here's the original: http://www.ebookworm.us/filtering-ms-word-text/

That being said, it looks like it's more for filtering output. I'd be more inclined to avoid it for now, first seeing if the other UTF-8 changes help. Also note that you can specify in any html form tag what character set should be used, which may help (depending on how much the browser respects it):

<form action="example.php" method="post" accept-charset="UTF-8">

[schwim]

Fantastic. I've got my gameplan:

1) change output to include the following on any user-inputted content:

htmlspecialchars($myVariable, ENT_QUOTES,'UTF-8');

2) Change forms to include the 'accept-charset' element.

Not done- Hold off on the function until I know whether what was done is sufficient.

I'm off to alter!

[laserlight]

Write a wrapper function, then should you change your mind, the alternative could be implemented with much less work.

[schwim]

Does something like this look safe to use?

/* Preparing content display for proper encoding and quote conversion */
function displayPrep($string, $enc, $quotes) {
	$result = '';

/* Allowed encs.  Currently, only one. */
if($enc == 'UTF-8'){
	$enc = 'UTF-8';
}else{
	/* No valid enc was passed. They get the default */
	$enc = 'UTF-8';
}

$string = stripslashes($string);
if($quotes = '1'){
	$string = htmlspecialchars($string, ENT_QUOTES, $enc);
}else{
	$string = htmlspecialchars($string, ENT_NOQUOTES, $enc);
}
return $result;
}

[Bonesnap]

Personally I would change this:

if($enc == 'UTF-8'){
    $enc = 'UTF-8';
}else{
    /* No valid enc was passed. They get the default */
    $enc = 'UTF-8';
}

...to this:

$enc = 'UTF-8';

...since $enc will always be assigned 'UTF-8' regardless.