Character encoding help

phpdino

Hi, I'm a newbie with this issue of character encoding and I have scoured the net for weeks trying to find a solution to this problem. I am trying to make sure that all text inserted into the db is actually stored as utf-8. I'm using mysql 5.1.11 and I even created a test database with one table. Charset utf8 Collation utf8_general_ci. Simple form submits to a php page. UTF-8 is declared everywhere but when I submit various languages like Greek, Turkish, Spanish, Asian fonts, etc. If I insert Español Türkçe Ελληνικά and then check it in query browser the first two show up and the greek word (third one) shows up as a bunch of vertical bars. When I check under the binary tab the characters that show up are messed up. Asian fonts are causing havoc too. Any ideas if I'm doing somthing wrong? Any help would be greatly appreciated. Thanks!

johanafm

Does your php script contain this?
header('content-type: type/subtype; charset=utf-8');

It doesn't have to be utf-8, but it must be an encoding that supports the characters you want to use. And actually you do not have to specify this header either, but if you don't, you will have to specify the accept-charset attribute in the form element

html 4.01 spec wrote:
accept-charset = charset list [CI]
This attribute specifies the list of character encodings for input data that is accepted by the server processing this form. The value is a space- and/or comma-delimited list of charset values. The client must interpret this list as an exclusive-or list, i.e., the server is able to accept any single character encoding per entity received.
The default value for this attribute is the reserved string "UNKNOWN". User agents may interpret this value as the character encoding that was used to transmit the document containing this FORM element.[/quote]
Do note the use of the word MAY in:
User agents may interpret this value as the character encoding that was used to transmit the document containing this FORM element.

But afaik, they do.

Next up is making sure the DB knows that you are talking to it uinsg utf-8. In mysql, it's dealt with by character_set_client, character_set_connection and character_set_results, all of which can be set with
SET names 'utf-8'
See http://dev.mysql.com/doc/refman/5.1/en/charset-connection.html

phpdino

I have this in the header

<? header('Content-Type: text/html; charset=utf-8'); ?>

but when I add your suggestion,
header('content-type: type/subtype; charset=utf-8'); it opens a dialog box to download the php file 🙁

It looks like everything is covered on the database side and I thought I had everything set on the web server side. The only other thing is the editor that is being used to insert conent. Tinymce. According to their configuration it should be right too but someone has to be wrong :S

johanafm

phpdino;10939449 wrote:
I have this in the header

<? header('Content-Type: text/html; charset=utf-8'); ?>

This is correct, and it has type: text; subtybe: html.
Do you send this header both when presenting the form and when presenting data retrieved from the db?

phpdino;10939449 wrote:
header('content-type: type/subtype; charset=utf-8'); it opens a dialog box to download the php file 🙁

This is commonplace functionality for user agent's when they do not reqcognize the content type specification. There is no type named type, and no subtype named subtype. The reason I did not specify them is that you might be using something other than text/html, such as application/xhtml+xml.

phpdino;10939449 wrote:
It looks like everything is covered on the database side

What makes you reach that conclusion? Just because the db stores the information as utf-8 doesn't mean it returns it to you as utf-8. Have you tried utf8_encode on the data returned from the db?

phpdino

johanafm;10939648 wrote:
This is correct, and it has type: text; subtybe: html.
Do you send this header both when presenting the form and when presenting data retrieved from the db?

This is commonplace functionality for user agent's when they do not reqcognize the content type specification. There is no type named type, and no subtype named subtype. The reason I did not specify them is that you might be using something other than text/html, such as application/xhtml+xml.

What makes you reach that conclusion? Just because the db stores the information as utf-8 doesn't mean it returns it to you as utf-8. Have you tried utf8_encode on the data returned from the db?

I'll give that a try

NogDog

A collection of what I know about the subject:
http://www.charles-reace.com/blog/2008/10/03/utf8-in-php-and-mysql/