getting title from web page.

bagus

Hi there,

I'm trying to do some 'scraping' I guess it's called.

First thing I want to do is to get the title of a web page.

Should be simple. I guess I'm having problems figuring out DOMDocument or something, but I've been stuck here for 5 hours..

So check this code. You can see I'm getting the phpbuilder home page, and can extract all the links from it and all the images, but I can't figure out how to just get the darned title. Down at the end you can see I've done it borrowing some regex stuff, but really it seems like I should have it already somehow...

$ch = curl_init('http://phpbuilder.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
//echo $curl_scraped_page;

$ss = substr($webpage,25,25);


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $burl);

curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$str = curl_exec($ch);
curl_close($ch);
$doc = new DOMDocument();
$doc->loadHTML($str);


$hrefs = $doc->getElementsByTagName('a');
$srcs = $doc->getElementsByTagName('img');
$titles = $doc->getElementsByTagName('title');

echo 'Amount of Links found:'.count($hrefs).'<br/>';
echo 'Amount of Sources found:'.count($srcs).'<br/>';

foreach($hrefs as $href)$matches_href[]= $href->getAttribute('href');
foreach($srcs as $src) $matches_src[] = $src->getAttribute('src');
//foreach($titles as $title) $matches_title[] = $title->??? ; //how can I just get the content of the title tag?

echo nl2br(var_export($matches_href,true)).'<br/>';
echo nl2br(var_export($matches_src,true)).'<br/>';




$str = stristr($str, '<title>');
$rest = substr($str, 7);
$extra = stristr($str, '</title>');
$titlelen = strlen($rest) - strlen($extra);
$gettitle = trim(substr($rest, 0, $titlelen));
echo "<br>Coming from: " . $gettitle . "<p>";

So any help appreciated. I'm definitely in the early stages of picking up php. Should I be using some sax parser or other html parser? Is this a reasonable approach? I guess I'll also want to grab things like the link text along with the href and I may need to sift thru some of the text on the page as well. I'm just using phpbuilder.com for this example. Where should I have gone to find the answer to this question on my own?

Thanks,

Bagus

stolzyboy

I would read the file in (can be done as a whole or line by line) and then you regular expressions to get the contents in the <title> tag

http://us.php.net/file
http://us3.php.net/pcre

bagus

Thanks, but this is the kind of solution I'm trying to avoid getting into.

For one, I have to think there are some methods out there that will parse an html page for me (and handle some bad html if necessary). Two I'm not that great at regex's to parse a whole page for all those tags attributes, etc.

I was trying to get it thru the use of something more like
getElementsByTagName as my example shows I have done for img and a tags.

Also you may note that I did include a 'regex' method of getting the title in my code... just to show that 'yah, I know you could do it that way, but there must be a better ...less manual.. way of doing it'.

Thanks

stolzyboy

There is nothing manual about regex. That's going to be your best bet, likely.

stolzyboy

There are some classes out there that'll parse HTML, if you want to go that route.

http://search.yahoo.com/search?p=php+html+parser&fr=yfp-t-501-s&toggle=1&cop=mss&ei=UTF-8

And, an FYI, you aren't using any regex in your code that I can see.

Horizon88

$ch = curl_init('http://phpbuilder.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
$pattern = '/\<title\>(.*)/i';
preg_match($pattern, $curl_scraped_page, $matches);
echo $matches[1];

returns:
"PHPBuilder.com, the best resource for PHP tutorials, templates, PHP "

There's probably a better regex for that. But this works.

Weedpacket

//foreach($titles as $title) $matches_title[] = $title->??? ; //how can I just get the content of the title tag?

Like you said, it's the content of the tag; it's a (text) node in its own right.

$title->textContent

bagus

Thanks a lot, weedpacket. That was the answer I was looking for.

I also now found that this would work too:

  echo "<br>Title text content: ".$title->nodeValue;