Why not use UTF-8?

Anon

Summary: I am making a multilingual site so why shouldn't I use UTF-8 encoding on all my pages when it works? If I use UTF-8, then why convert odd characters into entites?

For months on end, I have been working on a community based site, built on mysql using php with the Sablotron extention. XML/XSLT has been a great experience, and I don't regret for a second chosing it.

The site will eventually be in around 10 different language (the most important are english, french, chinese, japanese and korean). Some pages should be able to have text in several different characters sets (eg. be able to see japanese, english and chinese on the same page).

1) What encoding should I use?

I have tested everything with UTF-8, and it works like a charm. I can seen any number of different characters sets on the same page, and text I send using forms, is stored correctly and displayed corretly when retrieved. Ie. it works.

The thing is, UTF-8 is only relatively recently supported by browsers, and if I view the site in Windows encoding (ie. iso-8859-1) I can browse fine, but all the foreign chars are displayed as '??????'. This I could live with, but the worst problem is that when users submit data in a form, there is a good chance Sablotron will barf at the sight of special chars like æ,ï,$ etc. I cannot get around this by converting the string to entities, as I am assuming that they are viewing at UTF-8, and htmlentities() takes encoding as a parameter. So...

2) Is there a reliable way of knowing what encoding the user is currently using?

However, when browsing with UTF-8 I can view both ø AND ø without any problems. So...

3) IF I can assume it's safe to use UTF-8, why use htmlentities() at all to convert odd chars?

One solution could be to use iso-8859-1 for the lating character set languages (eg. english and french), and require utf-8 on all the odd character set languages (eg. chinese).

I have tested the site with the newest IE, Mozilla and Opera browsers without any problems. They detect UTF-8 automatically and show all the character sets fine (actually Opera had some problems displaying Korean). My guess is that about 3-5% of my users use browsers which don't support UTF-8 at the moment.

Any comments, links or answers?

Thanks in advance,
Jens

Anon

1) Definitely UTF-8. As for forms and data entry, that could be somewhat tricky indeed. We tested it for our project about half a year ago and only browser which could pass UTF-8 from forms was IE (5.x). I haven't tested it with latest Mozilla or Opera incarnations, though.

I haven't tried UTF-8 with Sablotron but in theory you could parse the input before giving it to Sablotron and wrap such entities in CDATA. Although that would be pretty cumbersome.

However, IIRC there was a way to handle those special chars nicely in XML without breaking it for parsers. My XML skills are a bit rusty so I can't really recall details. But dig around in XML sites and newsgroups, you might find something.

2) I assume you want to know not the charset user is currently using but rather whether his client accepts UTF-8? Well, browsers do tell you that with "Accept-Charset". My Mozilla, for example, reports happily among other things: "Accept-Charset: ISO-8859-1, utf-8". So you could check that.

3) Because its nice and correct way to do it. It's not really playing nice mixing charsets or coding characters otherwise than stated by the particular charset.

Anon

Thanx for the answers 🙂

I assume you want to know not the charset
user is currently using but rather whether
his client accepts UTF-8?

No, I meant, if I know that the client is currently browsing with UTF-8 encoding, then I can show everything without problems. IF I know that he is using iso-8859-1, then I would, for example, have to handle entities differently (eg, with form submitted data). So is there a way of knowing this, perhaps with javascript?

Well, browsers do tell you that
with "Accept-Charset". My Mozilla, for
example, reports happily among other
things: "Accept-Charset: ISO-8859-1, utf-
8". So you could check that.

Nice tip, I will look further into that.

Because its nice and correct way to do it.

Yes, I kinda figured that. Just the annoying thing, if I want to add a "send to email" function, I will have to convert the strings back to their original state (ø -> ø), which is not currently supported for UTF-8, but only for iso-8859-1. Oh well, guess I can code my way around that, somehow.

Cheers,
Jens

Anon

Checked accept-charset and it works fine with Mozilla, but neither IE or opera send it. Any ideas for others ways of detecting the accept charsets and/or current charset?

Jens