displaying english and chinese characters on the same pages

perpetualshaun

I have a long-standing project that has just recently started taking some foreign language submissions, and I'm having problems displaying the English and the Chinese special characters side-by-side.

The majority of the text on the page is in English, and when I tried converting the encoding on the whole page to UTF-8 (from ISO-8559-1), it showed question marks all over the place for some of the special characters that were pasted in from Word. So I began looking around on Google for a solution to convert just the text stings that are the Chinese characters to UTF-8, but nothing seems to be working.

I have tried the iconv, htmlentities, and htmlspecialchars, and combinations of the methods, and nothing's working. Here's what I have a the the moment:

html_entity_decode($response[$comments['comment_lbl']], ENT_QUOTES , 'UTF-8')

blackhorse

The submission may not be in utf8, it might be in big5 for traditional Chinese (Taiwan, HongKong etc.), and GB such as GB2312 for simplified Chinese (Mainland China).

perpetualshaun

For my site, as I'm pulling the chinese characters from MySQL, UTF-8 definitely will work. If I change the Content-Type (meta http-equiv="Content-Type" content="text/html; charset=utf-8), I see all the Chinese characters just fine.

johanafm

You claim that utf-8 works, so what's the problem?

But since using a meta tag to set character encoding is not to be relied upon, I suggest applying the appropriate header.

header('Content-type: text/html; charset=utf-8');

perpetualshaun

The problem is that when I set UTF-8 for the entire page, it shows some question marks for the special characters that were pastes in from Word (ndashes, quotes, etc.). That is the majority of the page. So I don't want to worry about having to fix those to gain the ability to view the Chinese characters. Isn't there a way to do both?

johanafm

Oh, now I see. Do you have the problematic word characters in the same column as the chinese or are they stored elsewhere?

If they are not in the same column, then you could try this:
If you get an error with those characters, and they are stored in a column using utf-8, then something went wrong when storing them. I'm not sure what happens with word copy-paste, but you could try doing that again, saving the results in a file instead of to DB, then open the file in a text editor that lets you specify the encoding to use. When it displays as it should, you know the encoding used. After that you can get the problematic data, convert it as needed and update the db.

If they are stored in the same column, you will require more work to fix it. You could use ord() to see for what values (ansii only) it fails, or try the uniord() or ordUTF8() posted as examples on http://nz.php.net/manual/en/function.ord.php then convert those values into their proper utf-8 character counterparts.

perpetualshaun

Yes, they are in the same column (field) in the database. These are comments for an online survey. Some of the comments have been pastes in from Word (hence some of the weird special characters), and others were entered directly through the website by the participants; in English, Chinese and some Japanese replies as well coming very soon.

I looked at the examples on this link, http://nz.php.net/manual/en/function.ord.php. I don't see how that's a solution for what I'm looking to do.

johanafm

What id does is tell you the byte value for a character. Ord treats each byte as one character, while utf-8 is encoded using 1 to 3. Or if it was 1 to 4. See wikipedia if you need the definitive specs.

Now, let's say you have $str = "Some ? strange? ?? chars"

$out = "";
for ($i = 0, $len = strlen($str); $i < $len; ++$i) {
	$out .= "$i: " . ord($str[$i]) . "\t" . $str[$i] . PHP_EOL;
}
file_put_contents("out.txt", $out);

0: 83		S
1: 111		o
2: 109		m
3: 101		e
4: 32		
5: 63		?
6: 32		
etc...

ord might not work perfectly since you're dealing with utf-8, but it should still give you an idea. What you are looking for is byte sequences that needs to be replaced. One ? could actually be encoded with several bytes.
Still, once you see what characters are supposed to be shown and have their byte sequences, write code to retrieve all data from the DB, loop over all rows and reconstruct the string while replacing the screwed up byte sequences with whatever is appropriate.

But if you don't have mixed working and non-working content in a single row, it might be enough to detect the problems, then simply doing an encoding conversion on the problematic rows and updating them.