[RESOLVED] XML Parsing

savj14

So here is the deal, I have a website with just under 1,000 Users. I found an API that pulls Xbox Live Info as an XML File. What I am trying to do is create a Cache System and store this information in my Database.

Currently the script does work, but it pulls the data from the API and stores the data very slowly. I get somewhere around 75 users before the script times out.

I am wondering if there is a better way to go about doing this. I'd like to have the process run quickly and run it as a corn job every 15 minutes or so if possible.

Before I paste my script here is a real basic example:

If you goto this site:
http://duncanmackenzie.net/services/GetXboxInfo.aspx?GamerTag=Festive+Turkey
I am parsing the following:

<Valid>true</Valid>
<Info>Last seen 12 hours ago playing Xbox 360 Dashboard</Info>
<Info2/>
<LastSeen>2008-04-22T04:03:43-05:00</LastSeen>
<Online>false</Online>
<StatusText>Offline</StatusText>
<Title>Xbox 360 Dashboard</Title>

I will save it to my database and use it later on.

Here is my script.

<?php

//ini_set('max_execution_time', 120);
//#!/hsphere/local/bin/php -q
set_time_limit(0);

$host="localhost";
$user="database_username";
$pass="databasepassword";
$database="database_name";
/* Crontabbed Stats */
function haloapi($contents, $get_attributes=1) { 
    if(!$contents) return array(); 

if(!function_exists('xml_parser_create')) { 
    //print "'xml_parser_create()' function not found!"; 
    return array(); 
} 
//Get the XML parser of PHP - PHP must have this module for the parser to work 
$parser = xml_parser_create(); 
xml_parser_set_option( $parser, XML_OPTION_CASE_FOLDING, 0 ); 
xml_parser_set_option( $parser, XML_OPTION_SKIP_WHITE, 1 ); 
xml_parse_into_struct( $parser, $contents, $xml_values ); 
xml_parser_free( $parser ); 

if(!$xml_values) return;//Hmm... 

//Initializations 
$xml_array = array(); 
$parents = array(); 
$opened_tags = array(); 
$arr = array(); 

$current = &$xml_array; 

//Go through the tags. 
foreach($xml_values as $data) { 
    unset($attributes,$value);//Remove existing values, or there will be trouble 

    //This command will extract these variables into the foreach scope 
    // tag(string), type(string), level(int), attributes(array). 
    extract($data);//We could use the array by itself, but this cooler. 

    $result = ''; 
    if($get_attributes) {//The second argument of the function decides this. 
        $result = array(); 
        if(isset($value)) $result['value'] = $value; 

        //Set the attributes too. 
        if(isset($attributes)) { 
            foreach($attributes as $attr => $val) { 
                if($get_attributes == 1) $result['attr'][$attr] = $val; //Set all the attributes in a array called 'attr' 
                /**  :TODO: should we change the key name to '_attr'? Someone may use the tagname 'attr'. Same goes for 'value' too */ 
            } 
        } 
    } elseif(isset($value)) { 
        $result = $value; 
    } 

    //See tag status and do the needed. 
    if($type == "open") {//The starting of the tag '<tag>' 
        $parent[$level-1] = &$current; 

        if(!is_array($current) or (!in_array($tag, array_keys($current)))) { //Insert New tag 
            $current[$tag] = $result; 
            $current = &$current[$tag]; 

        } else { //There was another element with the same tag name 
            if(isset($current[$tag][0])) { 
                array_push($current[$tag], $result); 
            } else { 
                $current[$tag] = array($current[$tag],$result); 
            } 
            $last = count($current[$tag]) - 1; 
            $current = &$current[$tag][$last]; 
        } 

    } elseif($type == "complete") { //Tags that ends in 1 line '<tag />' 
        //See if the key is already taken. 
        if(!isset($current[$tag])) { //New Key 
            $current[$tag] = $result; 

        } else { //If taken, put all things inside a list(array) 
            if((is_array($current[$tag]) and $get_attributes == 0)//If it is already an array... 
                    or (isset($current[$tag][0]) and is_array($current[$tag][0]) and $get_attributes == 1)) { 
                array_push($current[$tag],$result); // ...push the new element into that array. 
            } else { //If it is not an array... 
                $current[$tag] = array($current[$tag],$result); //...Make it an array using using the existing value and the new value 
            } 
        } 

    } elseif($type == 'close') { //End of tag '</tag>' 
        $current = &$parent[$level-1]; 
    } 
} 

return($xml_array); 
} 
	$connection=mysql_connect("$host", "$user", "$pass") or die("Failed to connect to mysql server");
		mysql_select_db("$database",$connection) or die("Database Error: Could Not Connect");
			$q=mysql_query("SELECT user_id,user_username FROM se_users");

	while($row=mysql_fetch_array($q)) {
/* Get the stats from bungie ere */
			$timestamp=time();
			$page = file_get_contents("http://duncanmackenzie.net/services/GetXboxInfo.aspx?GamerTag=".urlencode(trim($row['user_username'])));
			$stats = haloapi($page,0);
			//echo $row['user_username']."<br/>";
			if(!$stats) {
				break; /* To Break Out Of The Script and not kill the cpu in a loop!! */
			}


//mysql_query();
$update="INSERT INTO `xbl_status` 
(
`user_id`,
`gamertag`,
`valid`, 
`info`, 
`info2`,
`status_text`,
`country`,
`timestamp`
)

VALUES 
(
'".$row['user_id']."',
'".$stats['XboxInfo']['Gamertag']."',
'".$stats['XboxInfo']['PresenceInfo']['Valid']."',
'".$stats['XboxInfo']['PresenceInfo']['Info']."',
'".$stats['XboxInfo']['PresenceInfo']['Info2']."',
'".$stats['XboxInfo']['PresenceInfo']['StatusText']."',
'".$stats['XboxInfo']['Country']."',
'".$timestamp."'
)
";
mysql_query($update);
//mysql_query("UPDATE `se_users` SET `last_update`='$timestamp' WHERE use_id='$row[user_id]'");	
	}
?>

I also attached the database structure, just in case. With a sample a sample user.

savj14

Anyone?

sneakyimp

That duncan mckenzie site is slow as christmas...that is probably your problem. I don't think you'll be able to speed that up any at all. The best you can hope for is to be able to either a) fetch the xbox data yourself or b) disable the timeout function for PHP. disabling the PHP timeout is not always possible (like when safe mode is on, etc.) but is done thusly:

// disable time limit
set_time_limit(0);

savj14

I have the time limit set to zero in the script, but doesn't seem to help much.

I was under the impression that in order to get the Xbox Live Data myself I would have to be a part of the Xbox Live Development group. Which I don't believe they accepting new members.

I'd love to get the data myself If its possible....Any thoughts.

savj14

If this is possible(not sure if it is), would it be quicker to do the following:

1.) Grab the XML files from Duncans site
2.) Save the XML Files on my server
3.) Either take those XML files and save what I need to my database(parsing locally I would assume)

I would assume it would be much quicker to save 1,000 XML files on my server, than trying to parse real time.

Is this possible?? Or quicker??

NogDog

I might be inclined to create a separate script for updating your database from the remote site's data, and have it run periodically via cron. Then the data update process would be separated from your users' site access.

savj14

NogDog wrote:
I might be inclined to create a separate script for updating your database from the remote site's data, and have it run periodically via cron. Then the data update process would be separated from your users' site access.

So are you saying to do what I stated above....Save a separate XML File for each user?

If how would go about this? Since I have it setup to loop through the User's and goto Duncan's site. I assume I just need to add some code and stop the script from updating my database. And create the second script to handle that?

NogDog

Not necessarily. I'm suggesting storing the data in a database, as that is probably the most efficient way to access it dynamically. But instead of obtaining and parsing the source XML data every time a user accesses the site, have a separate script run periodically via cron to grab the data from the remote site, parse it, and insert/update it in the the database.

Now, if you want the data to be up-to-date to the minute, this might not be any help. But if you just need it updated every few hours or once a day, it might be worthwhile to consider.

savj14

Yeah I want to have it up to date at least every 15 minutes, possibly more frequent if possible.

I was doing some research and came across something. It will goto Duncans site and save the XML File to a cache directory on my server.

The only thing is, it doesn't grab everything properly. If I can get this working I would like to see how quickly it could loop through my users and save XML files on my server.............and then figure out how to take those file in my cache folder and insert them into my database.

<?

$gamertag = "ManHands14";

$filename = "cache/$gamertag.xml";
$url = "http://duncanmackenzie.net/services/GetXboxInfo.aspx?GamerTag=$gamertag";
$age = 3600;
if (file_exists($filename) && ((time() - filemtime($filename)) < $age)) {
    $usedfile = true;
    echo 'From Local File' . '<br />';
} else {  

    $f_contents = file_get_contents($url);
    $fp = fopen($filename, 'wb');
    fwrite($fp, $f_contents);
    fclose($fp);
    $usedfile = false;
    echo 'From Remote File' . '<br />';
}

//$xml = simplexml_load_file($filename);
//echo $xml->channel->title, '<br />';
//echo '<br />';
echo htmlspecialchars(file_get_contents($filename));

//foreach ($xml->channel->item as $item) {
  //  echo "<A HREF=\"$item->link\">$item->title</A>", '<br />';
    //echo $item->description, '<br />';
    //echo $item->pubDate, '<br />';
    //echo '<br />';
//}

if (!$usedfile) {
    echo $f_contents . '<br />';
    echo 'File Saved' . '<br />';
} 
?>

Keep in mind I don't have PHP5 yet so I cannot use simplexml, so I commented those portions out. If you run the script on your server you will notice that it does save an xml file to the cache directory. But some of the xml is missing or the format is not correct/parsing.

savj14

Ok so I have made some updates to my script and it now loops through my User Database, goes to Duncans site, writes the XML file to the Cache folder on my server.

Right now I am looping through only 10 Users, but the problem is that it still seems pretty slow.

Here is my updated script:

<?

//Database Info
$host="localhost";
$user="database user";
$pass="password";
$database="database";

//Not sure if this helps or not
set_time_limit(0);

//Get Gamertags from Database
$connection=mysql_connect("$host", "$user", "$pass") or die("Failed to connect to mysql server");
		mysql_select_db("$database",$connection) or die("Database Error: Could Not Connect");
			$q=mysql_query("SELECT user_id,user_username FROM users WHERE user_id > 0 AND user_id < 11");

//Loop Through the Database and assign Gamertags
while($row=mysql_fetch_array($q)) {
			$gamertag = $row['user_username'];
			//Find Spaces and replace them with %20
			$gamertag = str_replace(" ","%20",$gamertag);

//Save the XML per each Usertag to the Cache folder
$filename = "cache/$gamertag.xml";

//Duncans API  which contains the Xbox live info as an XML File
$url = "http://duncanmackenzie.net/services/GetXboxInfo.aspx?GamerTag=$gamertag";

//Grabbing the XML Contents, Opening and writing the File   

	$f_contents = file_get_contents($url);
	$fp = fopen($filename, 'wb');
	fwrite($fp, $f_contents);
    fclose($fp);
//For testing purposes
echo 'Done';
}
?>

Typically the File sizes are 13k. Is there a way to limit fwrite so it only writes 1k and then moves on to the next file? That might help speed things up.

laserlight

Is there a way to limit fwrite so it only writes 1k and then moves on to the next file?

Yes. Read the PHP manual concerning the third (and optional) parameter of [man]fwrite/man.

savj14

laserlight wrote:
Yes. Read the PHP manual concerning the third (and optional) parameter of [man]fwrite/man.

I didn't even see that when I looked at it the first time. Thanks!

Seems to help a little but it will screw up the XML Format, so I don't know if that will help.

Anyone have a suggestion for making it run quicker?

savj14

Frustrated

savj14

So I am able to get the POST using Curl

savj14

How would I go about cleaning this up(parsing?), taking what I need and updating my database?

Can someone guide me in the right direction?

sneakyimp

Your POST isn't very clear at all. You say you can get the POST using cURL. That sounds helpful. What is the format that gets returned? Are you going to have to scrape it? If so, [man]preg_match[/man] is your friend. If it's XML, your prior approach was probably the right way. The reason the other approach was slow was that the source website was slow - I'm sure the XML parser worked plenty fast.

savj14

The Source that is returned is HTML.

I'd like to create an XML Page from the Parsed HTML is at all possible, but one step at a time.

Here is curl code in action:

Removed link

So what I want is to be able to parse out the Gamertag, Offline/Online, and the Country for each entry. And then update my database, so the parsed info would have to be organized some how.

sneakyimp

Scraping a page is a pretty tedious business. You'll need to use [man]preg_match_all[/man] to do some pattern matching. If you come up with the right regular expression, you might isolate the region of the page that you are interested in.

For instance, assuming the content of that page is in a var called $html, you might isolate the table containing the users like this:

$pattern = '#<table class=\"dottedhorizontal\"[^\<]*>.*</table>#is';
$num_matches = preg_match_all($pattern, $html, $matches);

if ($num_matches == 0) {
  echo ' NO MATCHES FOUND';
} else {
  echo $num_matches . ' FOUND<br>';
  print_r($matches);
}

Once you have isolated that table, you could do additional matching to extract the repeating data.

This makes me wonder where that other site gets their data. Perhaps xbox live has some kind of data feed or api or rss or something?

savj14

Thanks....I'll have to try the preg_match out and see what I come up with.

One other Issues I cam across for curl. I am trying to loop through my database, grab the usernames and pass them through the POST

while($row=mysql_fetch_array($q)) {
/* Get the stats from bungie ere */
			$timestamp=time();
			$gamertag = $row['user_username'];
			$gamertag = $gamertag.',';
			//$gamertag = str_replace(" ","+",$gamertag);
			//$gamertag = $gamertag.'%2C%0D%0A';





$ch = curl_init('http://www.url_removed_for_example');
 curl_setopt ($ch, CURLOPT_POST, 1);
 curl_setopt ($ch, CURLOPT_POSTFIELDS, "postvalues");
 }
 curl_exec ($ch);
 curl_close ($ch);

When I do this it only passes the last usename through as $gamertag

If I move curl_exec($ch) right below curl_setopt ($ch, CURLOPT_POSTFIELDS, postvalues"); it will pass all usernames through but will do a seperate post for each one.

Any thoughts?

savj14

anyone?