UTF/ISO encoding problems

h4x0rmx

I'm working on an intranet application developed by someone else, I'm just fixing some bugs.
The application had a WYSIWYG editor that for some time used to encode the text in UTF-8 format, so now I randomly have some data UTF encoded on the database when everything else is ISO encoded.
Is there a way to just run a query and fix the problem (get rid of all those weird Â characters)?
If anyone has a suggestion, I'll really appreciate your help.

NogDog

I don't know off-hand of a way to do it at the database level, but if you have the [man]mbstring[/man] extension enabled in PHP, you could try retrieving the data, running it through the [man]mb_convert_encoding/man function, the updating the row with the converted text.

h4x0rmx

Would I have to run thought the whole database? It's pretty big!
While I was testing, I used something like this, but it wasn't working correctly, maybe I was doing something wrong.

if( mb_detect_encoding($str,"UTF-8, ISO-8859-1")=="UTF-8" )
{
$str iconv("UTF-8", "ISO-8859-1", $str);
}

Any suggestions?

NogDog

I would just try:

$str = mb_convert_encoding($str, 'ISO-8859-1');

Of course, the other option would simply be to output your PHP pages as UTF-8, as I believe all ISO-8859-1 characters should still output OK. But I'm by no means a character set expert, so I may be wrong.

h4x0rmx

Outputing my php pages as UTF-8 would seem like a good logical answer, but I tried it and it doesn't always work. Sometimes the pages that were ISO encoded will show weird characters.

NogDog

Just a thought: do you by any chance filter the output through htmlentities()? If so, when outputting the page as UTF-8, you should use htmlspecialchars(), instead, as the other character entities do not work well with UTF-8.

Piranha

Some special characters are part of ISO-8859-1 but not part of UTF-8. Characters like the swedish åäö and other western european characters. Those need to be converted to the entity number or entity name.

Using htmlspecialchars() won't convert those characters, htmlentities would do a better job according to the manual.

NogDog

Piranha;10879683 wrote:
Some special characters are part of ISO-8859-1 but not part of UTF-8. Characters like the swedish åäö and other western european characters. Those need to be converted to the entity number or entity name.

Using htmlspecialchars() won't convert those characters, htmlentities would do a better job according to the manual.

All I know is I needed to output a page as UTF-8, and when I used htmlentities() I got lots of place-holder characters, but when I used htmlspecialchars(), everything was perfect. Unfortunately, it's been awhile and I don't have access to the source of that site any more, so I'm not sure of all the specifics of that situation. 🙁 If anyone has a link to a good, concise, and readable reference explaining all this sh...stuff, I for one would be interested.

h4x0rmx

NogDog;10879682 wrote:
Just a thought: do you by any chance filter the output through htmlentities()? If so, when outputting the page as UTF-8, you should use htmlspecialchars(), instead, as the other character entities do not work well with UTF-8.

I don't use htmlentities at all, but I think it would be good to check if the WYSIWYG editor uses them anywhere.
Thanks for the suggestion.