XML parsing. XPath?

[leatherback]

Hi All,

I am (still) trying to efficiently parse many complex XML documents.

An example XML here:
http://www.hodexer.nl/hodex/uu/hodex_nl_uu_aw.xml Note that the XL files are very flaxible, so not each node is always present.

I cannot for the life of me figure out how to get this done. I did write a function to do this but it misses out whenever a group exists twice, only to be identified by differences in a conatined node or a language label. I am now thinking to use the xpath to retrieve individual nodes, as I do not need all information. But now I cannot figure out how to get this to work.

I am trying something along these lines:

function get_programme_info3($xmlUrl)
  {
  $dom = new DomDocument();
  $dom->load($xmlUrl);
  $xpath = new DomXPath($dom);

  $elements = $xpath->query('tuitionFee');

  if(!is_null($elements)) 
    {
    foreach ($elements as $element) 
      {
      echo "<br/>[". $element->nodeName. "]";

  $nodes = $element->childNodes;
  foreach ($nodes as $node) 
    {
    echo $node->nodeValue. "\n";
    }
  }
}

die();
//  return($out);
  }

But I get nothing.

The xml parser I was working with for the full file used:

function ReadNestedAttributedXML($xml, $last=null) {

$tree = null;
while($xml->read())

    switch ($xml->nodeType) {

case XMLReader::END_ELEMENT: return $tree;
    case XMLReader::ELEMENT:

  if($xml->hasAttributes)
    {
    $field 	= $xml->name;
    $xml	->moveToFirstAttribute();
    $AttName 	= $xml->name;
    $AttValue 	= $xml->value;
    $xml	->moveToElement();
    $tree[$xml->name][$AttName][$AttValue] = $xml->isEmptyElement ? '' : ReadNestedAttributedXML($xml);
    }
  else
    {


        if(is_array($tree) && isset($tree[$xml->name]))
                {

	    if(is_array($tree[$xml->name]) && array_key_exists('0',$tree[$xml->name]) )
	      {
                    $tree[$xml->name][] = ReadNestedAttributedXML($xml);
	      }
	    else
	      {
	      $tree[$xml->name] = array(0=>$tree[$xml->name]);
                  $tree[$xml->name][] = ReadNestedAttributedXML($xml);
	      }
                }
                else
                {
                    $tree[$xml->name] = ReadNestedAttributedXML($xml);
                } 


	    }

    break;

    case XMLReader::TEXT:

    case XMLReader::CDATA:
            $tree .= $xml->value;
break;
    }
return $tree;
} 


function ReadNestedAttributedXML_back2($xml, $last=null) {

$tree = null;
while($xml->read())

    switch ($xml->nodeType) {

case XMLReader::END_ELEMENT: return $tree;
    case XMLReader::ELEMENT:

  if($xml->hasAttributes)
    {
    $field 	= $xml->name;
    $xml	->moveToFirstAttribute();
    $AttName 	= $xml->name;
    $AttValue 	= $xml->value;
    $xml	->moveToElement();
    $tree[$xml->name][$AttName][$AttValue] = $xml->isEmptyElement ? '' : ReadNestedAttributedXML($xml);
    }
  else
    {
    if(is_array($tree) && array_key_exists($xml->name, $tree))
      {
      if(is_array($tree[$xml->name]))
        {
	$tree[$xml->name][] = $xml->isEmptyElement ? '' : ReadNestedAttributedXML($xml);
	}
      else
        {
	$tree[$xml->name] = array($tree[$xml->name], $xml->isEmptyElement ? '' : ReadNestedAttributedXML($xml));
	}
      }
    $tree[$xml->name] = $xml->isEmptyElement ? '' : ReadNestedAttributedXML($xml);
        }

    break;

    case XMLReader::TEXT:

    case XMLReader::CDATA:
            $tree .= $xml->value;
break;
    }
return $tree;
} 




function ReadNestedAttributedXML_back($xml) {

$tree = null;
while($xml->read())
    switch ($xml->nodeType) {
        case XMLReader::END_ELEMENT: return $tree;
        case XMLReader::ELEMENT:

            if($xml->hasAttributes)
			  {
			  $field = $xml->name;

			  $xml->moveToFirstAttribute();
			  $AttName = $xml->name;
			  $AttValue = $xml->value;

			  $xml->moveToElement();
			  $tree[$xml->name][$AttName][$AttValue] = $xml->isEmptyElement ? '' : ReadNestedAttributedXML($xml);
			  }
			else
			  {
			  $tree[$xml->name] = $xml->isEmptyElement ? '' : ReadNestedAttributedXML($xml);
              }

        break;

        case XMLReader::TEXT:
        case XMLReader::CDATA:
            $tree .= $xml->value;
    }
return $tree;
} 

[Weedpacket]

The "Path" in "XPath" is worth keeping in mind: it describes a path through the document starting from your current location (what the XPath specification calls the "context node") and finishing up wherever it is you're wanting to go. What makes it complicated is that (unlike a filesystem path) there might be several directions in which the next step in the path might take you, so you really end up with a tree instead of just a path.

For the concrete example given:

  $elements = $xpath->query('tuitionFee'); 

I'm assuming the intention is to get all the <tuitionFee> elements in the document. In that case the XPath expression is [font=monospace]//tuitionFee[/font].

It starts with [font=monospace]/[/font], making it an absolute path (i.e., starting from the document root).

[font=monospace]//[/font] is short for "the current node(s) and their descendants" - all nodes that are descendants of the current nodes including the current node itself. In this example there is only one "current node" and that's the document root, so we're talking about every node in the entire document.

[font=monospace]tuitionFee[/font] is short for "those child elements of the current nodes that are named 'tuitionFee'". At this point every node in the entire document is a current node, including every element that is the parent of a <tuitionFee> element. So the result is every <tuitionFee> element in the document.

[leatherback]

Hi WeedPacket,

Thank you for the reply. Indeed. In the example given I was trying to get one field, the tuition fee and see which tuition feel levels there are in the document. Adding the slashes did not give me any output. I have as good as no experience in XML processing, making this more dificult; I have no idea how PHP is accessing this.

If I use DOM manipulation, I can get things to work now;

function get_programme_info3($xmlUrl)
  {
  echo $xmlUrl."<br />";

  $dom = new DomDocument();
  $dom->load($xmlUrl);

  // try something else
  $params = $dom->getElementsByTagName('tuitionFee');
  $k=0;
  foreach($params as $fee)
    {
	$amounts = $params->item($k)->getElementsByTagName('amount');
	$j = 0;
	foreach($amounts as $amount)
	  {
	  echo "<br />TuitionFee: ".$amounts->item($j)->nodeValue;
	  $j++;
	  }
	$k++;
	}
  }

This results in:

<br />http://www.hodexer.nl/hodex/uvt/hodex_nl_uvt_eb.xml<br />
<br />TuitionFee: 1672
<br />TuitionFee: 1972
<br />TuitionFee: 1972
<br />TuitionFee: 1713
<br />TuitionFee: 1713
<br />TuitionFee: 6546

However, that seems like a horrible way of doing it, if you are wanting to retrieve some 50 fields from the XML, and have a several hundreds of XML files to retreive, I think it will be fairly slow? Is this worry justified?

Is there a way to do this more efficiently, and code it in an efficient manner?

As said, in the end I will need a few dozen fields from the XML. I am going to put this all in one class 'ProgrammmeInformation', which should contain all the functions to retrieve the information from the XML. Would I need to create a function for each main node that I want to retrieve? Or would there be a clear way to recycle the function?

For example:

The tuitionFee node contains a number of nodes which I would need:
- amount
- tuitionFeeRate
- tuitionFeeType
- programForm

And then there might be a node LanguageRequirement, with 5 occurances each containing 2 fields to retreive:
languageExam
minimumScore

[Weedpacket]

The biggest delay would be accessing and reading hundreds of XML files; compared to that the time spent processing each one is likely (read, I would expect it) to be small.

Obviously, what you'd want to do is load and parse each file once, and query its DOM multiple times; so the parsed DOM would be a property of your ProgrammeInformation class (if you're using XPath, so would its object).

The advantage of XPath would be its conciseness, and the fact that once it's parsed, the DOM traversal it implies would all happen in C-written code, rather than PHP. If its use doesn't exploit those advantages then it's no gain (getting a list of tuitionFee elements, for example: getElementsByTagName() is much more direct).

Whether or not there is a clear way to recycle one function for several tasks, you can find that out while you write it: write individual functions for each task; when you find yourself repeating the same code over and over, abstract that repetitive structure.

As for the XPath: what's needed is namespace registration (it's a part of the specification that's really borked: if the XML document uses namespaces you have to use them, but in that situation XPath has no concept of default namespace - you have to be explicit).

function tuitionFee_amounts($dom)
{
	$path = new DOMXPath($dom);
	$path->registerNamespace('x', $dom->lookupNamespaceUri($dom->namespaceURI)); 

$matched_nodes = $path->query('//x:tuitionFee/x:amount');
$length = $matched_nodes->length;
$matches = array();
for($i = 0; $i < $length; ++$i)
{
	$matches[] = $matched_nodes->item($i)->textContent;
}
return $matches;
}

That works for me (after I realised why it wasn't, and put that namespace registration stuff in). Two things to notice:

1. Really, the only things that would change to make that apply to each of your examples are the names of the elements appearing in the XPath expression.

2. After the $path has had the default namespace registered (and assigned to a prefix, sigh), the $dom object itself is not used again; so I could have created $path and passed that to the function instead, which then would only need the stuff after the blank line.

[leatherback]

Hi Weedpacket,

That is really helpfull, and probably the reason why I could not get xpath going. Thank you. I did not specify namespaces, but they are really abundant in de files I am using (If I understand the meaning of namespace correctly; I am not sure; I find documentation to be really minimal on this). As you were responding, I actually just started a new thread on the namespace problems I have.

-> I do not understand the query you just posted in the example. x is the registered namespace to be used I think. How would that relate to the language namespace used in the file (xml:lang="en")?

I have been able to push it into a fairly straightforward function, naturally without any error trapping in there, just rough "Why don't you * work" code 😉. later I will reshape the working setup to a class. not sure whether that is the best order to work in, but I can at least get to focus on the coding of the actual data first without wondering wether the class is defined properly.

The basic function is now:

// one example file:
$xmlUrl = http://www.utwente.nl/master/hodexdata/?get=BK

function get_programme_info4($xmlUrl)
  {
  // ensure right URL
  echo $xmlUrl."<br />";

  // Load the document
  $dom = new DomDocument();
  $dom->load($xmlUrl);

  // get tuition fee
  $TuitionInfo = get_node_info($dom, 'tuitionFee', array('amount', 'period', 'programForm', 'year'));

  // Get programme info
  $ProgrammeInfos = get_node_info($dom, 'programDescriptions', array('programName', 'programDescription'));
  }

function get_node_info($dom, $NodeName, $fields)
  {

  // http://www.php.net/manual/en/domdocument.getelementbyid.php

  $out = array();

  $params = $dom->getElementsByTagName($NodeName);

  $k=0;

  foreach($params as $p)
    {
    echo "<p>";
    $sub = array();

foreach($fields as $field)
  {
  echo $field;

/* The following line works. However, when multiple languages are defined, this is typically not the right one. So try to get namespaces involved. Then however it fails.
*/

//      $nodes = $params->item($k)->getElementsByTagName($field);

  $nodes = $params->item($k)->getElementsByTagNameNS('xml:lang="en"', $field);
  $sub[$field] = $nodes->item(0)->nodeValue;
  echo "<br />".$sub[$field];
  }
echo "</p>";  
$k++;  
}
  } 

[Weedpacket]

leatherback wrote:

-> I do not understand the query you just posted in the example. x is the registered namespace to be used I think. How would that relate to the language namespace used in the file (xml:lang="en")?

Nothing, really. Like I said, the XPath spec is broken here.

<program xmlns="http://www.hodex.nl" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.hodex.nl http://www.hodex.nl/hodex.xsd">

The base element specifies two namespaces to be used in the document: "http://www.hodex.nl" and "http://www.w3.org/2001/XMLSchema-instance". The latter is given a prefix, "xsi", so that uses of elements and attributes from it can be identified as such (e.g., "xsi:schemaLocation"). The former is the default - it's the one being used unless otherwise stated. Otherwise you'd need to prefix every element and attribute to identify which namespace applies.

But... what XPath sees is that every element and attribute of the document belongs to a namespace - that this attribute is from the "http://www.w3.org/2001/XMLSchema-instance" namespace and that element is from "http://www.hodex.nl" - but it isn't bright enough to make use of the same prefixes (or lack of prefixes) that are used in the document.

So it has to be told (the namespace has to be "registered" with XPath). And XPath isn't smart enough to allow registering a namespace using the absence of a prefix. You have to tell it that "xsi" is a prefix for "http://www.w3.org/2001/XMLSchema-instance", say, and you have to tell it that the prefix for "http://www.hodex.nl" is .... something. I picked 'x' arbitrarily.

getElementsByTagNameNS('xml:lang="en"', $field)

But "xml:lang" is the name of an attribute, not an element; defined in a reserved namespace with the reserved "xml" prefix.

If - for the sake of example - you wanted all Dutch webLinks, the XPath expression would be [font=monospace]//x:webLink[@xml:lang='nl'][/font] (where again I needed to indicate which namespace "webLink" belonged to).

[leatherback]

Excellent help weedpacket. I am now through the main battle. Would be happy to see pointers on how to make this more efficient / effective. But I think it is pretty straightforward now. I will pull all the $path definitions out of the funtion too, as this function will be called several times in a row on the same xml file. I think that would reduce the processing time..?

function get_node_info4($dom, $NodeName, $fields='', $language ='')
  {
  $path = new DOMXPath($dom);
  $path->registerNamespace('x', $dom->lookupNamespaceUri($dom->namespaceURI));
  $out = array();

  // Form xpath query
  $q = "//x:".$NodeName;

  // Add language component
  if($language <> '')
    {
	$q .= '[@xml:lang = "'.$language.'"]';
	}

  // grab nodes
  $matched_nodes = $path->query($q);

  // Test output
  $length = $matched_nodes->length;
  if($length == 0)
    {
	return(-1);
	}

  // process nodes   

  for($i = 0; $i < $length; ++$i)
    {
	if(is_array($fields))
	  {
	  // Sub-nodes have been defined. Grab them

  $sub = array();
  foreach($fields as $field)
    {
    $nodes = $matched_nodes->item($i)->getElementsByTagName($field);
    $sub[$field] = $nodes->item(0)->nodeValue;
	}
  $out[] = $sub;
  }
else
  {
  // no subnodes defined. Grab only the values from the main nodes
  $out[] = $matched_nodes->item($i)->textContent;
  }
}
  return $out;
  }