The correct way to deal with different character sets?

TheoGB

Okay so I've built my CMS and I've discovered it has issues with certain characters, namely things like 'smart' quotes and apostrophes like word makes and é.

Someone submitted a piece of text that included these but when I load it from the database it won't load into the edit pane. I can run the same request on phpMyAdmin and I can see the body text is being brought back so it's there.

Equally when I display it on screen I get those annoying ?s in black diamonds showing up. This doesn't happen with other articles with smart quotes in etc. UTF8 Decode also obviously gave some odd results since my routine to strip out words for my search and kill apostrophes etc failed on this.

The bottom line is that I'm clearly doing something wrong. I can remove these special characters from the document in phpMyAdmin, then load it up in my CMS edit screen then re-apply them and they then show up okay!

So, what sort of things are we talking about going wrong, here?

http://londonprogressivejournal.com is the site if you want to check the HTML header information.

NogDog

This might help: http://www.charles-reace.com/blog/2008/10/03/utf8-in-php-and-mysql/

johanafm

I have two additions to NogDog's url.

I'd preset the DB to use utf-8 for everything, thus removing the need to specify this for each new table or field, and also remove the need to set names 'utf8'.
Also, in your html head element include a meta element that matches your content type header:
```
<meta http-equiv="content-type" content="text/html; charset=utf-8"> (or ending in /> if using xhtml)
```
The reason is that IE doesn't care about character set information in the http header but do consider the meta element. At least up until IE7, but I'm rather certain it goes for IE8 as well.

TheoGB

Okay, cheers guys.

Looking at it, it's definitely some kind of freaky UTF-8 issue. If I parse out my form like this:

$SmartyObject->assign("$key", $value);

I get the text but the UTF-8 symbols show up as the weird inverted ?s.

If I do this:

$SmartyObject->assign("$key", htmlentities($value, ENT_QUOTES, 'UTF-8'));

I get nothing at all.

I'll check out NogDog's link and see if that helps me solve it.

NogDog

johanafm;10959548 wrote:
I have two additions to NogDog's url.

I'd preset the DB to use utf-8 for everything, thus removing the need to specify this for each new table or field, and also remove the need to set names 'utf8'.

Good point, though I'm not sure (i.e. ignorant) about situations where you're using a shared host as to whether you can change the default connection configuration?

2. Also, in your html head element include a meta element that matches your content type header:
<meta http-equiv="content-type" content="text/html; charset=utf-8"> (or ending in /> if using xhtml)
The reason is that IE doesn't care about character set information in the http header but do consider the meta element. At least up until IE7, but I'm rather certain it goes for IE8 as well.

In the "wearing suspenders and a belt" spirit, it's probably a good idea to always send the header() and set the meta tag. 🙂

sneakyimp

Char encoding can be a confusing concept. I have found that it helps to make sure that EVERY SINGLE STEP in your website should declare itself as utf8 and you must use the multibyte string functions instead of the regular string functions. This means:

HTML pages that have input forms should declare themselves as UTF-8, not Latin-1 or anything else.
Your PHP that handles form submissions should use the multibyte string functions. Otherwise, you get weird string length values because é--obviously just one char--is encoded using two bytes in UTF-8 and only one byte in Latin 1
Your MySQL database, tables, and connection should be declared UTF-8 whereever you have the option.

And just like all those input stages, the output stages should be declared UTF-8.

I realize this sounds like a tall order and it is. You may have trouble with Smarty templates not being UTF-8 friendly. Another alternative is to use Latin-1 encoding for everything, but then you are out of lucky for any asian languages and possibly cyrillic too. It'll work for most Western languages though.