Reading the results of a search but limiting what is displayed

spivey

I have created a search window that accesses content from a company we subscribe to. The site provides content related to books, including cover art, in addition to a lot of other stuff that I don't want to display. And from the page that is called up, one could access administrative stuff for that site.

Here is my form:
<form method="POST" action="http://contentprovider'swebsite/searchpage.html">
<input type="text" size="30" name="keyword" value="" />
<input type="submit" size="5" name="Submit" value="GO" />
<input type="hidden" size="30" name="landing_page" value="content_list" /></form>

I would like to only get the cover art, the "src=http://contentprovider'swebsite/image.jpg" but all that would result from a particular search.

I have used "file_get_contents" on the "searchpage.html" and I get the blank page's contents returned.

In order to get the results of a search, would I put some php in the <form....action=""> element? Or, how can I read the output and display only the elements that I want?

Thanks
spivey

sneakyimp

You should look into using [man]cURL[/man]. cURL is a library that lets PHP act as a browser or ftp client or whatever -- it can fetch pages and files and gives you lots of control.

Once you have determined how to fetch the remote page's contents, you'll need to parse that HTML for the images you are looking for.

I don't really see any other way to do it -- unless the remote company lets you specify more options about what you want to see on those pages.

scrupul0us

curl is definitely overkill for this...

I would pull the page into PHP's DOM function and walk the DOM for the elements you want

--or--

Regex the whole page for the elements you want

spivey

Would I do this as part of an "action" statement, because using the site's URL as the action statement returns an external page?

sneakyimp

If you don't use cURL, how are you supposed to POST and retrieve the contents of the search result from the remote server?

spivey

So, for the URL, would I basically build the URL search form data
"http://contentprovider'swebsite/searchpage.html?keyword=&Submit=GO&landing_page=content_list"

except with the $_POST variables, and do all of this in a .php document that would be called via the "Action" element?

sneakyimp

spivey;10886088 wrote:
So, for the URL, would I basically build the URL search form data
"http://contentprovider'swebsite/searchpage.html?keyword=&Submit=GO&landing_page=content_list"

No. Appending the various search parameters to the search page's URL would be correct if the search engine accepted parameters through GET. Your original form example uses POST. Using cURL, you can post a form on somebody else's site from within PHP:

$post_fields = array(
  'text' => 'your search text',
  'landing_page' => 'content_list',
  'Submit' => 'GO'
);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://contentproviderswebsite/searchpage.html'); // set the remote url
curl_setopt($ch, CURLOPT_POST, 1); // yes we are posting
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields); // this is our POST data
curl_setopt($ch, CURLOPT_HEADER, 0); // no headers in output
curl_setopt($ch, CURLOPT_VERBOSE, 1); // verbose output, good for debugging
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);// $ch will return the results of your POST when you execute
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"); // the remote sites often check for a known user agent

$result = curl_exec ($ch);
curl_close($ch);
echo "<h1>Here is your result</h1>";
echo htmlspecialchars($result);

or something like that

spivey

Thanks for that. The original form uses "POST" (I had to switch to "GET" using the FF "Developer" plugin to get the URL).

Would I use htmlentities() to gather only the "HTML entities" that I would want to display?

Or, would I echo only the pieces that I want?

Thanks a lot.

spivey

sneakyimp

you should read the documentation on [man]htmlentities[/man]. It probably doesn't do what you think it does.

The curl code I put up will fetch the remote page and return its HTML contents as a string. As I said in my original post:

sneakyimp wrote:
Once you have determined how to fetch the remote page's contents, you'll need to parse that HTML for the images you are looking for.

To parse it, I would see what kind of things uniquely identify the images I'm after and try to use [man]preg_match[/man] to pick them out.

scrupulous recommended using some thing like this to parse the document.

spivey

I haven't worked with DOM before, so bear with me.

$doc = new DOMDocument();
$doc->loadHTML($result);

/*If I "print_r($result);" I get the results page as if I had entered terms from the content provider's search page. */


/* Attempting to identify HTML "<img" tags--not working*/
$doc->getElementsByTagName('img');

/*since the above stuff isn't working, I can't sort out the "preg_match" stuff yet.*/
$pattern = "<img src=\"http://contentprovider.images.com";/*this URL prefix is common to all image links, so I would like to match using it*/

$page_parse = preg_match($pattern, $page, $matches);
echo $page_parse[0]->saveHTML();

Could I also use "eregi" instead of "preg_match"?

Thanks

spivey

sneakyimp

the DOM approach sounds like it could work...I think you have to assign it to a variable:

$images = $doc->getElementsByTagName('img');
print_r($images);

for preg_match, you should try something like this:

$pattern = '#<img src="(http://contentprovider.images.com[^"]+)[^>*]>#i';
$matches = array();
$num_matches = preg_match($pattern, $result, $matches);
print_r($matches);

You should read the docs on [man]preg_match[/man] as the stuff that you find in $matches will vary depending on how many sets of parentheses you put in there. I think $matches[1] is what you'll be interested in.

spivey

<?php

$post_fields = array(
  'keyword1' => $_POST['keyword1'],
  'Submit' => 'GO',
  'binding' => null,
  'landing_page' => 'content_list'
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://contentproviderswebsite/searchpage.html'); // set the remote url
curl_setopt($ch, CURLOPT_POST, 1); // yes we are posting
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields); // this is our POST data
curl_setopt($ch, CURLOPT_HEADER, 0); // no headers in output
curl_setopt($ch, CURLOPT_VERBOSE, 1); // verbose output, good for debugging
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);// $ch will return the results of your POST when you execute
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"); // the remote sites often check for a known user agent

$result = curl_exec ($ch);
curl_close($ch);
echo "<h1>Choose your cover</h1>";
$doc = new DOMDocument();
$doc->loadHTML($result);


$images = $doc->getElementsByTagName('img');

echo "images print statement";
print_r($images);

$pattern = '#<img src="(http://contentprovider.images.com[^"]+)[^>*]>#i';
$matches = array();
$num_matches = preg_match($pattern, $result, $matches[1]);
echo "matches print statement";
print_r($matches);

This is what I tried, and it hasn't generated images. I get

images print statementDOMNodeList Object ( ) matches print statementArray ( [1] => )

As you can see, I have assigned the "$images" variable to the $doc->getElements... statement.

My question is, in the $num_matches lines, should why do you use $result? Does $result now carry the info from $images and/or $doc variables?

$result stands in for "$subject" according to the PHP manual, and is the input string. But has the value of $result changed?

Thanks

spivey

sneakyimp

It was my understanding from your post that $result contained the HTML of the page you were trying to parse. If $result contains that html, then your code should be like this:

$pattern = '#<img src="(http://contentprovider.images.com[^"]+)[^>*]>#i';
$matches = array();
$num_matches = preg_match($pattern, $result, $matches);
echo "preg match found " . $num_matches . " matches<br>";
print_r($matches);

You put $matches[1] in your preg_match call which is not correct. Read the docs on [man]preg_match[/man] and maybe you will understand. preg_match searches $result using the pattern specified in $pattern and puts all the matches it finds into $matches which should come out as an array. Like I said in my post, the structure of $matches will depend on your pattern -- especially what sort of parenthetical groupings you use.

spivey

I am not getting any matches. However, now I know that no matches are being made.

Question: If I am using $result, do I need all of this ?

$doc = new DOMDocument();
$doc->loadHTML($result);
$images = $doc->getElementsByTagName('img');

because if I

print_r($images);

, I get nothing.

spivey

sneakyimp

I am not familiar with DOMDocument and apparently print_r and the DOM classes are not super friendly to each other. I looked up a tutorial and, assuming you have fetched the remote page into $result, then this might work for you:

$doc = new DOMDocument();
$doc->loadHTML($result);
$nodes = $doc->getElementsByTagName('img');

for($i=0; $i<$nodes->length; $i++) {
        echo 'node ' . $i . ' src=' . $nodes->item($i)->getAttribute('src') . '<br>';
}

I was also a bit sloppy with my pattern, but BE WARNED that I'm assuming a couple of things with this pattern:
1) all images have a src attribute that is surrounded by DOUBLE QUOTES
2) all images have a src attribute that starts with 'http://contentprovider.images'
assuming your html is correctly loaded into $result, then this might also work:

$pattern = '#<img src="(http://contentprovider.images[^"]+)"[^>*]*>#i';
$matches = array();
$num_matches = preg_match_all($pattern, $result, $matches);
print_r($matches);

spivey

I tweaked the echo statement to display the images.

Thanks, sneakyimp. By the way, could you point me to a good tutorial on regular expressions, so I could learn to make sense of your URL formula?

$pattern = '#<img src="(http://contentprovider.images[^"]+)"[^>*]*>#i';

Thanks again

spivey

sneakyimp

Glad it's working. Regular expression tutorial? I don't know. Google would probably now better than me. I learned a bit about them in college but have relied most heavily on the php documentation for my day-to-day experience.

This book on them is reviewed well:
http://oreilly.com/catalog/9781565922570/

spivey

The regexp tutorials i found online didn't quite do it.

I had just placed a hold on that book in my library system, so I'm looking forward to getting it.

thanks again

spivey

sneakyimp

Being good at regex takes a lot of practice. The biggest gotchas are:
1) You have to start and end your expression with / or # or one of a few other characters that will be accepted. I'm not really sure what all these characters are, but I use those two and switch them depending on what my pattern is looking for. if I'm looking for something with slashes, I use the #. Otherwise, I stick with /. I have no idea why your pattern must be surrounded by these chars. Unless perhaps it has something to do with the next gotcha.

2) You can follow the closing # or / with some modifiers that make the regex ignore case or recognize patterns across multiple lines. These are called modifiers. e.g., an 'i' means ignore case.

3) You have to learn which characters are meaningful as special regex characters. That documentation link I posted is useful for that. You may find [man]preg_quote[/man] helpful too.

Beyond those gotchas, it's really about practice. They're very powerful. You might want to put this regex testing form on your server. It's useful for testing expressions:

<?php

function strip_slashes_recursive($arg){
    if (is_array($arg)){
        $result                         = array();
        foreach($arg as $key => $value){
            $result[$key]               = strip_slashes_recursive($value);
        }
    } else {
        $result                         = stripslashes($arg);
    }
    return $result;
}

if (get_magic_quotes_gpc() == 1){
    $_POST = strip_slashes_recursive($_POST);
}

if($_POST['Submit']) {
	$pattern=$_POST['pattern'];
	if (!$pattern) {
		$error = "You didn't enter a pattern!";
	}
	$string=$_POST['string'];
	if (!$string) {
		$error = "You didn't enter a string to be searched!";
	}

if ($error) {
	show_form($error, $data);
} else {
 // do the pattern match

	$matches = Array();
	$num_matches = preg_match_all($pattern, $string, $matches);

	if ($num_matches == 0) {
		$error = "NO MATCHES";
		show_form($error, $_POST);
	} else {
		show_form('', $_POST, $matches);
	} // if matches found
}
} else {
	show_form();
}

function report_html3($str_arg) {
	echo "<pre style='background-color:#eeeeee;'>" . 
htmlentities($str_arg) . 
"</pre>\n";
}

function show_form($error='', $data=Array(), $matches=Array()) {
?>
<html>
<body>
<h1>Testing Regular Expressions</h1>
<?
	if ($error) {
		echo "<div style='color:#ff0000'>" . $error . "</div>\n";
	}
?>
<p>This form allows you to test a regular expression using <a href="http://php.net/preg_match_all">preg_match_all()</a></p>
<form method=POST>
String:<br>
<textarea name='string' cols='100' rows='10'><?=htmlentities($data['string']) ?></textarea><br><br>
RegEx Pattern:<br>
<input type=text name="pattern" size=100 value="<?= htmlentities($data['pattern']) ?>"><br><br>
<input type=submit value="Test Pattern" name="Submit">
</form>

<?
	$num_matches = sizeof($matches[0]);
	if ($num_matches > 0) {
		echo "<hr>";
		echo "<h2>" . $num_matches . " MATCHES</h2>";
		foreach($matches as $key1=>$matches1) {
			foreach ($matches1 as $key2=>$value) {
				echo "<b>Match " . $key1 . "-" . $key2 . "</b>\n";
				report_html3($value);
			} // for each match
		} // for MAIN MATCH
	}	// if matches
?>
</body></html>
<?
} // show_form()
?>

spivey

In the sources I looked at online, they used different characters, where you were using the #, so thanks for that explanation. Also about the "i", I didn't get that either.

I will put that form on my test site at work.

Thanks again for all the info!

spivey