XML Parsing Problem, Data Cleaning

[l008com]

Hello
I'm making a php XML parser that loads data into mysql database. I'm using just the basic XML functions. The files I'm parsing are uber huge, so I can't use one of the easier XML parsing methods that load the whole file into memory. So anyway, I read a file line by line, and pass each line into the parse. I keep getting invalid character errors. I use the utf8_encode() function and it got a little further. I added a function to replace all &'s with &amps; (which apparently is a bug in the XML file, that this isn't already done) and it got a little further. But its still not making it nearly all the way through the file. This particular is 256 MB that I'm trying to parse, though once this is squared away, I'm going to have an even bigger one I want to build a parser for to run once a week.

So what I really need, is a way to clean the xml data before giving it to the parser. With MySQL, i use the myql_real_escape_string() function before giving it data, and it works nicely. Is there any function or technique I can use to do the same kind of cleaning to lines of XML files? Even if it just deletes "illegal" characters?

[l008com]

Here's my script, FYI.
The XML file is a very simple format, its just very very long.

<?

set_time_limit(0); 

$insiderest = false; 

$tag = ""; 

$title = ""; 
$country = ""; 
$address = ""; 
$city = ""; 
$state = ""; 
$zip = ""; 
$phone = ""; 
$url = ""; 

// Create an XML parser 
$xml_parser = xml_parser_create(); 

// Set the functions to handle opening and closing tags 
xml_set_element_handler($xml_parser, "startElement", "endElement"); 

// Set the function to handle blocks of character data 
xml_set_character_data_handler($xml_parser, "characterData"); 

// Open the XML file for reading 
$fp = fopen("/path/to/zmlfile.txt","r") 
       or die("Error opening xml file."); 

// Read the XML file one line at a time 
while ($data = fgets($fp,4096))
	{
	//Clean String for XML parser
	$data = utf8_encode($data);
	$data = str_replace("&","&amp;",$data);

// Parse each line with the XML parser created above 
xml_parse($xml_parser, $data, feof($fp))
	// Handle errors in parsing 
	or die("XML error: ".xml_error_string(xml_get_error_code($xml_parser))."(".xml_get_error_code($xml_parser).") at line ".xml_get_current_line_number($xml_parser)."++$data++");
}

// Close the XML file 
fclose($fp); 

// Free up memory used by the XML parser 
xml_parser_free($xml_parser);



function startElement($parser, $tagName, $attrs)
	{
	global $insiderest, $tag;

if ($insiderest)
	{
	$tag = $tagName;
	}
elseif ($tagName == "RESTAURANT")
	{
	$insiderest = true;
	}
}

function characterData ($parser, $data)
	{
	global $insiderest, $tag, $title, $country, $address, $city, $state, $zip, $phone, $url;

if ($insiderest)
	{

	switch ($tag)
		{ 
		case "D:TITLE": 
			$title .= mysql_real_escape_string(trim($data)); 
			break; 
		case "COUNTRY": 
			$country .= mysql_real_escape_string(trim($data)); 
			break; 
		case "ADDRESS": 
			$address .= mysql_real_escape_string(trim($data)); 
			break; 
		case "CITY": 
			$city .= mysql_real_escape_string(trim($data)); 
			break; 
		case "STATE": 
			$state .= mysql_real_escape_string(trim($data)); 
			break; 
		case "ZIP": 
			$zip .= mysql_real_escape_string(trim($data)); 
			break; 
		case "PHONE": 
			$phone .= mysql_real_escape_string(trim($data)); 
			break; 
		case "URL": 
			$url .= mysql_real_escape_string(trim($data)); 
			break; 
		} 
	}
}

function endElement($parser, $tagName)
	{
	global $insiderest, $tag, $title, $country, $address, $city, $state, $zip, $phone, $url,$xml_parser;

if ($tagName == "RESTAURANT")
	{

	$query = "INSERT INTO `chefmoz_list` (name,country,address,city,state,zip,phone,url) VALUES('$title','$country','$address','$city','$state','$zip','$phone','$url')";
	mysql_query($query);
	echo "Insert restuarant `$title` into database [".mysql_insert_id()."]\n";

	$insiderest = false;
	$title = ""; 
	$country = ""; 
	$address = ""; 
	$city = ""; 
	$state = ""; 
	$zip = ""; 
	$phone = ""; 
	$url = ""; 
	}
}
?>

[Weedpacket]

Hint; [php]..[/code] can be used for syntax-highlighting your PHP code (as you can see in other posts).

I haven't looked too hard, but I do note that you don't specify UTF8 encoding when you create the parser.

[l008com]

According to the Docs, input encoding is automatically detected in PHP 5.
http://us.php.net/manual/en/function.xml-parser-create.php