[RESOLVED] XML Parsing

[savj14]

NogDog wrote:

I might be inclined to create a separate script for updating your database from the remote site's data, and have it run periodically via cron. Then the data update process would be separated from your users' site access.

So are you saying to do what I stated above....Save a separate XML File for each user?

If how would go about this? Since I have it setup to loop through the User's and goto Duncan's site. I assume I just need to add some code and stop the script from updating my database. And create the second script to handle that?

[NogDog]

Not necessarily. I'm suggesting storing the data in a database, as that is probably the most efficient way to access it dynamically. But instead of obtaining and parsing the source XML data every time a user accesses the site, have a separate script run periodically via cron to grab the data from the remote site, parse it, and insert/update it in the the database.

Now, if you want the data to be up-to-date to the minute, this might not be any help. But if you just need it updated every few hours or once a day, it might be worthwhile to consider.

[savj14]

Yeah I want to have it up to date at least every 15 minutes, possibly more frequent if possible.

I was doing some research and came across something. It will goto Duncans site and save the XML File to a cache directory on my server.

The only thing is, it doesn't grab everything properly. If I can get this working I would like to see how quickly it could loop through my users and save XML files on my server.............and then figure out how to take those file in my cache folder and insert them into my database.

<?

$gamertag = "ManHands14";

$filename = "cache/$gamertag.xml";
$url = "http://duncanmackenzie.net/services/GetXboxInfo.aspx?GamerTag=$gamertag";
$age = 3600;
if (file_exists($filename) && ((time() - filemtime($filename)) < $age)) {
    $usedfile = true;
    echo 'From Local File' . '<br />';
} else {  

    $f_contents = file_get_contents($url);
    $fp = fopen($filename, 'wb');
    fwrite($fp, $f_contents);
    fclose($fp);
    $usedfile = false;
    echo 'From Remote File' . '<br />';
}

//$xml = simplexml_load_file($filename);
//echo $xml->channel->title, '<br />';
//echo '<br />';
echo htmlspecialchars(file_get_contents($filename));

//foreach ($xml->channel->item as $item) {
  //  echo "<A HREF=\"$item->link\">$item->title</A>", '<br />';
    //echo $item->description, '<br />';
    //echo $item->pubDate, '<br />';
    //echo '<br />';
//}

if (!$usedfile) {
    echo $f_contents . '<br />';
    echo 'File Saved' . '<br />';
} 
?>

Keep in mind I don't have PHP5 yet so I cannot use simplexml, so I commented those portions out. If you run the script on your server you will notice that it does save an xml file to the cache directory. But some of the xml is missing or the format is not correct/parsing.

[savj14]

Ok so I have made some updates to my script and it now loops through my User Database, goes to Duncans site, writes the XML file to the Cache folder on my server.

Right now I am looping through only 10 Users, but the problem is that it still seems pretty slow.

Here is my updated script:

<?

//Database Info
$host="localhost";
$user="database user";
$pass="password";
$database="database";

//Not sure if this helps or not
set_time_limit(0);

//Get Gamertags from Database
$connection=mysql_connect("$host", "$user", "$pass") or die("Failed to connect to mysql server");
		mysql_select_db("$database",$connection) or die("Database Error: Could Not Connect");
			$q=mysql_query("SELECT user_id,user_username FROM users WHERE user_id > 0 AND user_id < 11");

//Loop Through the Database and assign Gamertags
while($row=mysql_fetch_array($q)) {
			$gamertag = $row['user_username'];
			//Find Spaces and replace them with %20
			$gamertag = str_replace(" ","%20",$gamertag);

//Save the XML per each Usertag to the Cache folder
$filename = "cache/$gamertag.xml";

//Duncans API  which contains the Xbox live info as an XML File
$url = "http://duncanmackenzie.net/services/GetXboxInfo.aspx?GamerTag=$gamertag";

//Grabbing the XML Contents, Opening and writing the File   

	$f_contents = file_get_contents($url);
	$fp = fopen($filename, 'wb');
	fwrite($fp, $f_contents);
    fclose($fp);
//For testing purposes
echo 'Done';
}
?>

Typically the File sizes are 13k. Is there a way to limit fwrite so it only writes 1k and then moves on to the next file? That might help speed things up.

[laserlight]

Is there a way to limit fwrite so it only writes 1k and then moves on to the next file?

Yes. Read the PHP manual concerning the third (and optional) parameter of [man]fwrite/man.

[savj14]

laserlight wrote:

Yes. Read the PHP manual concerning the third (and optional) parameter of [man]fwrite/man.

I didn't even see that when I looked at it the first time. Thanks!

Seems to help a little but it will screw up the XML Format, so I don't know if that will help.

Anyone have a suggestion for making it run quicker?

[savj14]

Frustrated

[savj14]

So I am able to get the POST using Curl

[savj14]

How would I go about cleaning this up(parsing?), taking what I need and updating my database?

Can someone guide me in the right direction?

[sneakyimp]

Your POST isn't very clear at all. You say you can get the POST using cURL. That sounds helpful. What is the format that gets returned? Are you going to have to scrape it? If so, [man]preg_match[/man] is your friend. If it's XML, your prior approach was probably the right way. The reason the other approach was slow was that the source website was slow - I'm sure the XML parser worked plenty fast.

[savj14]

The Source that is returned is HTML.

I'd like to create an XML Page from the Parsed HTML is at all possible, but one step at a time.

Here is curl code in action:

Removed link

So what I want is to be able to parse out the Gamertag, Offline/Online, and the Country for each entry. And then update my database, so the parsed info would have to be organized some how.

[sneakyimp]

Scraping a page is a pretty tedious business. You'll need to use [man]preg_match_all[/man] to do some pattern matching. If you come up with the right regular expression, you might isolate the region of the page that you are interested in.

For instance, assuming the content of that page is in a var called $html, you might isolate the table containing the users like this:

$pattern = '#<table class=\"dottedhorizontal\"[^\<]*>.*</table>#is';
$num_matches = preg_match_all($pattern, $html, $matches);

if ($num_matches == 0) {
  echo ' NO MATCHES FOUND';
} else {
  echo $num_matches . ' FOUND<br>';
  print_r($matches);
}

Once you have isolated that table, you could do additional matching to extract the repeating data.

This makes me wonder where that other site gets their data. Perhaps xbox live has some kind of data feed or api or rss or something?

[savj14]

Thanks....I'll have to try the preg_match out and see what I come up with.

One other Issues I cam across for curl. I am trying to loop through my database, grab the usernames and pass them through the POST

while($row=mysql_fetch_array($q)) {
/* Get the stats from bungie ere */
			$timestamp=time();
			$gamertag = $row['user_username'];
			$gamertag = $gamertag.',';
			//$gamertag = str_replace(" ","+",$gamertag);
			//$gamertag = $gamertag.'%2C%0D%0A';





$ch = curl_init('http://www.url_removed_for_example');
 curl_setopt ($ch, CURLOPT_POST, 1);
 curl_setopt ($ch, CURLOPT_POSTFIELDS, "postvalues");
 }
 curl_exec ($ch);
 curl_close ($ch);

When I do this it only passes the last usename through as $gamertag

If I move curl_exec($ch) right below curl_setopt ($ch, CURLOPT_POSTFIELDS, postvalues"); it will pass all usernames through but will do a seperate post for each one.

Any thoughts?

[savj14]

anyone?