Strange question

steveDD

In my admin area I have a text box where I can amend or add content to my website. It updates my mysql database. However when it is retrieved and laid out on the site it chnges the simple characters to squares for example

£ = �
' = �

NogDog

What character encoding are you using for the page? Are you using [man]htmlentities/man to filter the output?

Weedpacket

And what character encoding are you using in the database? I find that using a consistent character set through the whole process (database, page, even the editor) eliminates virtually all hassles like this. I use UTF-8 as a matter of personal preference.

nrg_alpha

Weedpacket;10882862 wrote:
And what character encoding are you using in the database? I find that using a consistent character set through the whole process (database, page, even the editor) eliminates virtually all hassles like this. I use UTF-8 as a matter of personal preference.

While I am not well versed in databases, the issue of character encoding strictly server dependant?

My site utilises iso-8859-1... but as an experiment, I changed my index page from this line:

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

to:

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

Well, let's just say that the code would not validate (w3c validator complains that the document cannot be checked). Is this due to confilct with live server configurations? Or even worse, bad code? Am I missing something?

EDIT:
The w3c validator error message is as follows:

'Sorry, I am unable to validate this document because on line 137 it contained one or more bytes that I cannot interpret as utf-8 (in other words, the bytes found are not valid values in the specified Character Encoding). Please check both the content of the file and the character encoding indication.

The error was: utf8 "\xE9" does not map to Unicode'.
The issue seems to be with regards to my Resumé tab.. .but I thought UFT-8 encompasses more characters than iso-8859-1

Weedpacket

Ah, but in UTF-8 é is encoded as "\xc3\xa9", not "\xe9".

nrg_alpha

Weedpacket;10882900 wrote:
Ah, but in UTF-8 é is encoded as "\xc3\xa9", not "\xe9".

Oh..I see.. I was under the understanding that UTF-8 maintained the same character encoding as iso-8859-1, but added some additional ones (meaning all the characters in iso-8859-1 would be the same in UTF-8).

I suppose not.

Weedpacket

nrg_alpha wrote:
I suppose not.

No - it wouldn't work, because one byte can only represent 256 characters, so if you want to represent the hundred-thousand-odd characters and formatting symbols of basic Unicode you will need to use some of those bytes to represent escape sequences to signal that they're part of longer multibyte codes. ISO-8859-1 pretty much claims all 256 bytes, leaving none for additional characters.

UTF-8 maintains backward compatibility with ASCII, in that every valid ASCII character is a valid UTF-8 character, but ASCII only occupies bytes 0x00..0x7f, leaving bytes 0x80..0xff to encode longer sequences. The codepoints for ASCII characters all lie beween 0x0000 and 0x007F inclusive, and map to one-byte UTF-8 character codes.

é (its official Unicode name is "LATIN SMALL LETTER E WITH ACUTE") is at codepoint 0x00E9. ISO-8859-1 maps that to "\xe9", while UTF-8 maps it to "\xc3\xa9". (There actually is a meaning behind that. There'd have to be, or it wouldn't be computable. Writing it in binary gives the sixteen bits 11000011-10101001. The bits in bold make 00011101001, which in hex is E9. The sequence starts with "110", so it's the start of a two-byte sequence, and the other byte starts with "10". Neither is therefore an ASCII character (which all start with "0"), the second byte can't be mistaken for the start of a multibyte sequence (which start with "110", "1110", etc.), and the first byte can't be mistaken for the continuation of one because they all start with "10".)

nrg_alpha

Thanks for the explanation, Weedpacket. It appears I have much to learn in many areas (including character encoding). I'll have to redigest this all 🙂

Cheers,

NRG

steveDD

guys you have just thrown me, erm I am new to this how do I find out what i am using. Lets take it back to basics 🙂

NogDog

In the <head> section of your HTML output, do you have a content-type meta tag, something like:

<meta http-equiv="content-type" content="text/html; charset=utf-8">

If so, what is the value where the above has "utf-8"?

steveDD

I have

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

NogDog

What character set does your database table and/or columns use. Also, are you using [man]htmlentities/man for any of the output in question?

steveDD

where do I find out the character set of the DB?
and also what is the later.

Very new to this.

Steve

illzz

You can find out the default character set for a database by running SHOW CREATE DATABASE.

You can see what the current tables' character sets are by running SHOW TABLE STATUS. You will want to look at the Collation column.

You may want to also run SHOW CHARACTER SET for a description of the Collations.