xml_parse and document encoding

[venutip]

Hey everyone. I'm having a problem parsing XML files in PHP 4.3.9.

I have a few files I'm trying to parse. I'm sure that they're all well-formed because I can view them without problems in both IE 6.0 and FF 1.0.6. When I go to parse them, though, xml_parse returns one of these errors:

1. invalid token
2. unclosed token
3. not well-formed (invalid token)

I did a little investigating and found that when the files are fresh (they are auto-generated by another product), and I open them in TextPad, the encoding is set to Unicode. But if I use TextPad to save them as UTF-8 documents, the errors disappear and the script works perfectly!

So, my questions are:

1. Why does that happen?
2. Is there any way I can use PHP to encode the docs as UTF-8 before I try to parse them? I tried reading the original XML file into a string, using utf8_encode to set it to UTF-8, and then writing it to a temp file, but that doesn't seem to work.

By the way: There is no way for me to adjust how the XML is generated. I have to work with these files as is.

Anybody have any ideas??? 😕

Thanks!

[Weedpacket]

Hmmm... a UCS2 to UTF8 translation .... utf8_encode only does ISO8859-1 to UTF8. There's a [man]recode[/man] extension that uses GNU Recode but it sounds like you're using a Windows system. [man]iconv[/man] exists for both; but as it happens, while I'm looking at that latter page I see there's code in the user notes that states explicitly that it's for translating UCS2 to UTF8. That may be a simpler approach than installing an entire extension for just one conversion.

[venutip]

Thanks! Iconv sounds like it will work. But I can't get it enabled on my Windows system. This is what the manual says:

Note to Windows® Users: In order to enable this module on a Windows® environment, you need to put a DLL file named iconv.dll or iconv-1.3.dll (prior to 4.2.1) which is bundled with the PHP/Win32 binary package into a directory specified by the PATH environment variable or one of the system directories of your Windows® installation.

So I copied iconv.dll and php_iconv.dll to my windows\system32 directory and my php directory. I also added php\extensions and php\dlls to my PATH. But still, when I try to run iconv(), I get a call to an undefined function. Not sure what I'm doing wrong.

Maybe it's just time I upgraded to PHP 5. Probably be a lot simpler. Anyway, if you have any ideas on why this still isn't working, I'd appreciate it.

[Weedpacket]

Well, I just noticed that the generic manual link I gave went to the chapter, rather than the iconv function itself. The page and note I was looking at was http://www.php.net/manual/en/function.iconv.php#49171

The input and output formats are a bit different - sequences of hex digits rather than actual characters - so the function would need to be modified so that it reads "\0x06\0x45\0x06\0x31\0x06\0x2D" (that's UCS2 - two bytes per character, where "\0x06" is a single byte) instead of "06450631062D"; and similarly for output.

So (not that I've tested it or even checked to see that it runs let alone is correct) something like

function ucs2toutf8($str)
{
	$utf8='';
	for ($i=0;$i<strlen($str);$i+=2)
	{
		$in_byte1 = $str{$i};
		$in_byte2 = $str{$i+1};
		if ($in_byte1 == chr(0)) // null "\0x00" byte
		{
			$utf8 .= $in_byte2;
		}
		else
		{
			$two_bytes = $in_byte1.$in_byte2;
			$out_byte1 = chr(192+intval(ord($two_bytes)/64));
			$out_byte2 = chr(128+(ord($two_bytes)%64));
			$utf8 .= $out_byte1.$out_byte2;
		}
	}
	return $utf8;
}