[RESOLVED] Grab Text From Webpage

burstroo

Hello, I'm trying to figure out what functions I would need to use in php to be able to go out to a webpage and grab some text. I'll figure out how to make it go to the website, etc at a later time - I am just curious on how I would actually grab text when I'm there.

Thanks

Naito

Going to the website and grabbing the text are one and the same.

Example:

<?php

$filetext = file_get_contents('http://us3.php.net/manual/en/print/index.php');
echo "Check it out. You just with your php file pulled the web page content from the web and posted it on you're site. Good job.<p> $filetext";

?>

burstroo

I think you misunderstood me. I understand how to grab the contents on the page, but I'm interested in text only. Say for example, if I wanted to go to my site and create some text, and then on my other website copy that exact same text (without my html, and all that stuff) .. with some php.

Naito

I see. That seems to be very similar to a problem that I was having. There are three basic ways that I'm aware of at the moment to pull it off. Option (1) know what exact line the text you want to copy from the page is on. Option (2) Use delmiters that are not used anywhere else on you're page to seperate it from the stuff before it and another at the end of it to seperate the stuff after it and then after getting the contents of the file to a variable expload the variable and use the 2nd array variable created. Option (3) When you create text in your one page copy it to blank datafile in the same directory, then when pulling it you can pull directly from that datafile. Need further examples?

Naito

Note, for info involving how to read only a specific line of a file go here - http://www.designdetector.com/2006/02/flat-file-database-demo-3.php

burstroo

That's kindof what I'm getting at. Although I can't say too much due to an NDA, but what I want to do is somewhat 'spider' the webpage and just take all the text that's visible to the human eye (not the html).

I will do this in my project:

Visit the site via my spider (This thread is about a function I will give my spider)
Take the text only from the web page. None of the <html><title> stuff ... I just want text that is visible via browser.
Do something to that text (Can't say due to NDA)
And print it somewhere ...

But .. what I can't figure out is how to do #2.

Naito

Sorry it took so long, I'm still working on it in fact but I have a theory. Try this out.

<?php

$urlpage = 'http://us3.php.net/manual/en/print/index.php';
$filetext = fopen("$urlpage", "r");
while (!feof($filetext))
{
$theline = fgetss($filetext, 2048);
echo "$theline<br>";
}
fclose($filetext);

?>

Naito

Think that is the kind of stripping you are talking about? Feel free to change the $urlpage variable to represent another site you might want to test on.

burstroo

Yes, this is 'exactly' what I am looking to do. 🙂 If you happen to improve it anyway let me know I would love to take a look at it. Im going to have to try and get it setup though to where I can say, hey don't look at the first .. xx characters, and don't look at the last xx characters.. (So I don't get the headers of the page, or footers.. just pretty much content.. ther'es probably no way of avoiding it 'exactly' but I can do my best.. thinks for this. 🙂

burstroo

Ah, i just realized it's getting rid of 'all' the html, except for the <br> tags. If I can get it to that stage I'm going to be set. 🙂 Thanks again man! You've been a life saver.

Naito

No problem. The key command in the whole thing is fgetss which is exactally the same as fgets except it strips all the html and php tags from the file in question. I'm new to PHP myself so thats the only reason it prints all the data right as it's processing it. I'm not sure how to create an array and add to the array with each loop. If you know how please forward that part of the code back to me.

burstroo

I'm not too sure on that one either. Do you think you might be able to understand how to not include css in that breakdown.. i.e everything between <style> and </style> ..

Example...

See... I used it against this blog..
http://m-i-l-d-e-w.blogspot.com/2006/04/your-lack-of-concern-overwhelms-me.html

and ... the css is included.. comments too.. Any ideas?

http://svcsports.net/blogged/

Naito

Yep I believe you can allow tags with a string. http://us2.php.net/manual/en/function.fgetss.php is the php manual's guide to the function.

So basically $theline = fgetss($filetext, 2048, '<style><a>');

Would allow the style tags and links. Note you only have to include the opening tag, closing tags are included automatically.

burstroo

We had a miscommunication I think. I don't want to allow those tags, but make sure I dont show what's in between the <style></style> tags.. the css isn't blocked.. just the html is removed away from it.

Naito

Yes, I see that..... Now I've got a few ideas on how to fix it, but I can't figure out how to fully utilize preg_replace.... I need it to replace a string of unknown length from the file. I'm going to keep trying.

burstroo

It's very appreciated! 🙂 Thanks again.

Naito

Alrighty... I've got it a little bit more cleaned up now.

<?php

$urlpage = 'http://m-i-l-d-e-w.blogspot.com/2006/04/your-lack-of-concern-overwhelms-me.html';
$pagedata = "";
$filetext = fopen("$urlpage", "r");
while (!feof($filetext))
{
$theline = fgets($filetext, 2048);
$pagedata .= "$theline<br>";
}
fclose($filetext);

$pagedata = ereg_replace("<head>(.)head>", "", $pagedata);
$pagedata = ereg_replace("<script>(.)script>", "", $pagedata);
$pagedata = strip_tags($pagedata, '<br>');
echo "$pagedata";
?>

Still not perfect, and now it's all jumbled together but it's better then it was. If you get it refined any more before I do post the code so I can see how, this is quite an interesting thing.

saraadmin

I don't know whether or not its of interest, but can i just draw your attention to PHP's excellent DOM implementation, and particularly its ability to make DOM trees out of HTML files: loadHTML()

Naito

I find it works a little bit better like this with <p> substituted for <br> since it keeps the paragraphs closer together.

<?php

$urlpage = 'http://m-i-l-d-e-w.blogspot.com/2006/04/your-lack-of-concern-overwhelms-me.html';
$pagedata = "";
$filetext = fopen("$urlpage", "r");
while (!feof($filetext))
{
$theline = fgets($filetext, 2048);
$pagedata .= "$theline<p>";
}
fclose($filetext);

$pagedata = ereg_replace("<head>(.)head>", "", $pagedata);
$pagedata = ereg_replace("<script>(.)script>", "", $pagedata);
$pagedata = strip_tags($pagedata, '<p>');
echo "$pagedata";

?>

By the way Saraadmin: It might be, if I could figure out how to get it working to pull the text out but so far I'm not having much luck.

burstroo

We're getting it there. Here's what I've got.

<?php

$urlpage = 'http://stardustholiday.blogspot.com/';
$filetext = fopen("$urlpage", "r");
while (!feof($filetext))
{
$theline = fgetss($filetext, 2048);
$theline = ereg_replace("<head>(.*)head>", "", $theline);
$theline = ereg_replace("<script>(.*)script>", "", $theline);
$theline = ereg_replace("<style language='text/css'>(.*)style>", "", $theline);

echo "$theline";
}
fclose($filetext);

?>

As you can see though, I still have quite a bit of css there from the style tags. (I've tried just <style> but that isn't working. Also, there's not a need for the <br> or the <p> tags.. This will eventually be placed into a db and converted to an algorithym that I have came up with, but I need to weed out all the css first.. and I've been unsuccessful a lot.