character encoding and seo optimization issue

sneakyimp

I've inherited a big nasty oscommerce project with a lot of dodgy modules installed. One of the many things I'm trying to get my head around is the way in which a product name (or category name, etc.) gets converted into an seo-friendly url.

The problem I'm having is that the product names in the database appear to use iso-8859-1 names which contain some unusual chars such as an ellipsis (ord=133). This not a particular problem for most display purposes, as the charset declared in the html is iso-8859-1. The problem appears to be this function which is supposed to convert a string of text into a string of words-and-dashes that can serve as an seo-friendly url. This function seems to be the culprit:

	function strip($string){
		if ( is_array($this->attributes['SEO_CHAR_CONVERT_SET']) ) $string = strtr($string, $this->attributes['SEO_CHAR_CONVERT_SET']);
		$pattern = $this->attributes['SEO_REMOVE_ALL_SPEC_CHARS'] == 'true'
						?	"([^[:alnum:]])+"
						:	"([[:punct:]])+";
		$anchor = ereg_replace($pattern, '-', mb_convert_case($string, MB_CASE_LOWER, "utf-8"));
		$pattern = "([[:space:]]|[[:blank:]])+"; 
		$anchor = ereg_replace($pattern, '-', $anchor);
		return $this->short_name($anchor); // return the short filtered name 
	} # end function

I'm not sure I see any wisdom in converting to a multibyte string while using single-byte ereg statements (and isn't ereg deprecated?). The net effect of this function is that the phrase "Something…and something else" becomes "somethingand-something-else". The ellipsis just disappears.

Removing the mb_convert_case function call and using strtolower helps (interestingly [[:punct]] fails to match the ellipsis), but I'm hoping to make this work even if folks are using kanji or arabic or whatever. I'm guessing I'll need to change the character encoding declaration of my pages at the very least.

Does anyone have some good general advice about making sure a website is utf-8 friendly? Some things occur to me

1) Data is captured via browser in form inputs. I'm guessing these pages where I capture user/admin input should declare a meta tag that specifies utf-8:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

2) any PHP code that processes this input should use the multibyte string functions. I was sad to notice that there are no preg functions in this collection, just ereg.

3) My databases and database tables should all be declared with utf-8 as the charset. I have no idea how this relates to (and possibly conflicts?) with the collation declared for a given column.

4) Any PHP code that processes database output should also use the multibyte string functions.

5) When the html (or xml or whatever) is finally blasted out to the browser, it should have the meta tag declaring said output as utf-8 (see item 1 above). Does it also need to send a header?

Any guidance or wisdom here would be much appreciated.

Weedpacket

sneakyimp wrote:
I was sad to notice that there are no preg functions in this collection, just ereg.

That's because the PCRE functions already support UTF-8 (the 'u' modifier).

johanafm

sneakyimp;10942067 wrote:
5) When the html (or xml or whatever) is finally blasted out to the browser, it should have the meta tag declaring said output as utf-8 (see item 1 above). Does it also need to send a header?

You should send that header. However, the meta content-type tag is not needed, and should for the most part be avoided. If you serve utf-8 content and provide no http conte-type header, but do provide meta content-type, IE8 will treat the document as something other than utf-8, possibly the user's desktop language settings.

Providing a header with with charset A and meta charset B results in the document be treated as charset A in most (all?) browsers.

sneakyimp;10942067 wrote:
My databases and database tables should all be declared with utf-8 as the charset. I have no idea how this relates to (and possibly conflicts?) with the collation declared for a given column.

You actually do not need to store data with any specific charset just because you want to retrieve it as such. Automatic translation can deal with this. You can also use one charset for the database db (default for all tables), another charset for the table db.t (default for all columns of table A) and a third for the column db.t.c.
But using utf-8 for everything is probably a good idea.

Charset specifies the encoding for the characters. That is, what byte sequence represents a specific character. Not handling this correctly produces garbage output.

Collation contains information about the characters and is language specific. For example, ORDER BY ASC for
a, ä, b, z
with collatin latin1_german1_ci would probably give you just that order, while latin1_swedish_ci would give you
a, b, z, ä
Not handling collation correctly will screw up your result ordering.

sneakyimp

Really good info guys!

Thanks Weedpacket! I want to say I am still profoundly grateful for your tutelage regarding multibyte character strings and such. The docs on preg say:

docs wrote:
This modifier turns on additional functionality of PCRE that is incompatible with Perl. Pattern strings are treated as UTF-8. This modifier is available from PHP 4.1.0 or greater on Unix and from PHP 4.2.3 on win32. UTF-8 validity of the pattern is checked since PHP 4.3.5.

It says pattern strings (as opposed to the search string). It's not exactly detailed or clear about what the u flag does. Do you know where I might find more info?

Johana: Thanks for the great insight. What do you mean by 'Automatic translaction' ? Also, would you agree that encoding and collation are entirely distinct or no? Seems to me that a collation of latin1_swedish_ci would be useless for sorting Kanji or Arabic.

sneakyimp

OH I nearly forgot.

I remember hearing that internationalized domain names are just around the corner (or possibly are already here?).

Given that I'm talking about urls in this post, I can't help but wonder what it means to have a url which contains non-ascii characters.

johanafm

sneakyimp;10942268 wrote:
It says pattern strings (as opposed to the search string). It's not exactly detailed or clear about what the u flag does. Do you know where I might find more info?

I'd recommend simple testing to find out exactly what it means. To me, the wording suggests that the pattern can either be a sequence of bytes, or a sequence of possibly multibyte characters.
Assume you have a two-byte character A in your pattern and a two-byte character B in your search string, encoded by bytes 0xI 0xJ and 0xI 0xK respectively, J != K. If you do consider the pattern as consisting of single-byte characters, The first half of A would match the first half of B, which may lead to "strange" results.
In order to test it, find two multibyte characters where some leading byte(s) matches, and match '#[A]#' against 'B' and then '#[A]#u' against B.

And indeed. Using ordUTF8() as defined in user comment at http://se.php.net/manual/en/function.ord.php, and uchr() at http://se.php.net/manual/en/function.chr.php

// Find the character following the euro sign
$a = '€';
$ordA = ordUTF8($a);
echo "ord($a): " . $ordA . '<br/>';
echo 'next: ' . uchr($ordA + 1) . '<br/>';

// Use one as search string, the other as pattern.
$string = '€';
$p = '#[&#8365;]#';

if(preg_match($p, $string, $m))
	echo '<pre>'.print_r($m,1).'</pre>';
else
	echo 'no match<br/>';

$p = '#[&#8365;]#u';

if(preg_match($p, $string, $m))
	echo '<pre>'.print_r($m,1).'</pre>';
else
	echo 'no match<br/>';

sneakyimp;10942268 wrote:
1. What do you mean by 'Automatic translaction' ?
2. Also, would you agree that encoding and collation are entirely distinct or no?
3. Seems to me that a collation of latin1_swedish_ci would be useless for sorting Kanji or Arabic.

1. character_set_client, character_set_connection, character_set_result: http://dev.mysql.com/doc/refman/5.1/en/charset-connection.html
2. Encoding and collation are semi-distinct. You cannot have an encoding of utf8 and a collation of latin1_swedish_ci. If you wish a swedish interpretation of the strings, you would have to use collation utf8_swedish_ci or encoding latin1.
3. Yes. But I believe you can specify the use of another collation than column defaultin queries. So using encoding_general_ci might be an idea and then collate as necessary in specific queries. Or keep kanji stored in one place, swedish in another.

sneakyimp;10942269 wrote:
OH I nearly forgot.

I remember hearing that internationalized domain names are just around the corner (or possibly are already here?).

Given that I'm talking about urls in this post, I can't help but wonder what it means to have a url which contains non-ascii characters.

slashdot wrote:
country-code Top Level Domains

Apart from this, iirc the URI scheme allready handles non-ascii. And furthermore there is supposed to be some kind of string normalization. Read the RFCs to see exactly what is allowed and how it works. Or should work. Even if they decided to allow full non-ascii domains, it will be a long while before every server around the world can deal with it.
Starting with http://en.wikipedia.org/wiki/Url and the references to RFCs found therein, as well as the page for URI linked to from the given link and its RFCs.