PROS! A simple, effective way to scrape an HTML page?

brianb

Hello people,

I know all the function to open a page and get it's conetnts, but pulling the actual data from the content that I want is where I am confused...

I've seen done a million different ways, but I'm looking for the simplest most effective way to do it...

Basically, I need some code to grab all the content in between to HTML TAGS.

For instance...

$start = " ";
$stop = "<hr noshade>";

$content = "The entire HTML page goes here... This is the content I want to grab<hr noshade>";

So, after my script grabs the contents of the page, and finds the $start and $stop, it'll return what's in the middle....

Make sense?

ANY and ALL help is MUCH appreciated! Thanks in advance!

WinterDragon

Try this...

<?php
$content = "The entire HTML page goes here...<br><br>This is the content I want to grab<hr noshade>";
#This will start the search at the begining...
$start = strpos($content, "<br><br>"); 
#You might need to change this to strrpos() because then it will search it backwards..
$stop = strpos($content, "<hr noshade>"); 
$chopedContent = substr($content, $start, $stop);
echo $chopedContent;
?>

Worked fine when I did it hehe... Should work just fine for ya 🙂

brianb

Hmmmm...

Ok, it's not stopping when it's supposed to....

Check out my script:

<?PHP

$url = 'http://www.amazon.com/exec/obidos/tg/detail/-/B000004286/';

   // Open the URL and grab contents 
   $file       = fopen ($url, "r"); 
   $contents   = fread ($file, 1000000);  // Read around 1 MB
   fclose($file);

//This will start the search
$start = strpos($contents, "<br /><br />");
$stop  = strrpos($contents, "<hr noshade size=1>");

$result = substr($contents, $start, $stop);

echo $result;

?>

Any ideas why it's not stopping? I think it might be that <hr noshade size=1> appears multiple times on the page.

I want it to STOP after the first instance.

Any further help is appreciated!

brianb

anyone have any ideas on how to get this working?

All help is appreciated! Thanks!

barand

What about

$start = strpos($contents, "<br /><br />");
$stop  = strpos($contents, "<hr noshade size=1>", $start);

which will find the first 'hr noshade size=1>' after the start position

brianb

Ah....a very good try, BUT still not working!

Any other ideas? Is this the best way to do this? Should I start over from scratch? 🙁

brianb

Anyone have any ideas on this?

I just wanna scrape! 🙂

superwormy

Crap if you give me an email address and wait until tommarow I'll send you a function to do this... don't have it here at work though.

Mine actually returns an array of matches, it was designed to find things like this:

this is regular text which we DONT want but we want this text but not this, stuff it isn't bold but this stuff willb e the second elelment in the array it returns

And the array looks like:

$array[0] = "but we want this text";
$array[1] = "but this stuff will be the second elelment in the array it reutnrs";

And you can set it to limit it to a certain number of results, so only return teh first match if you want... Send me an email postmaster@durdenwebapplications.com and I"ll email you it tommarow.

brianb

Hey Superwormy,

I shot you an e-mail...did you get it? I'd really love to know how to dow this!

Thanks!

superwormy

Hey yeah, I'll email it to you as soon as I get home, round 6:30ish, sorry for the delay!

brianb

Thanks wormy!!!

brianb

hey Wormy...

Your script is great, only I'm not getting something right here....

Can you give me an example of the usage?

superwormy

Sure...

Heres the string we have, let's assume we want to pull out all the bold text.

<?php

$string = " keith rules, he is the man! and we all really like him because he is so cool! ";

// note we put the string as the first arguement,
// the second argument is where we should start getting
// stuff, and the third is when we should stop
$array = Get_String_Between_Strings ($string, "", "");

// print out the array
print_r ($array);

This is what print_r() will print out:

Array
(
[0] => the man!
[1] => really like him
[2] =>
)

Note that the second, third and fourth arguments to the function are OPTIONAL, if you don't supply an arguement to the third and fourth, it will default to < and >, and will return an array of everything not inside of HTML tags. The fourth once simply slices the array that gets returned, so if you only wanted to get back ' the man! ' in the above example, just set the fourth argument to 1, and you'll only get that one result. LMK if you have more questions...

brianb

Hey Wormy...ok...we're getting closer (I think)...still stuck though...

Now do I edit ANYTHING in the actual function OR do I just set my variables accordingly when I call the function?

Here's what I'm using as my function call:

$tracks_table = Get_String_Between_Strings($contents," ","<hr noshade size=1>", 1);

$contents is equal to the ENTIRE PAGE that is being read. (This might be messing up what's in the function? I'm not sure)

Here's an example page I'm trying to get data from:
http://www.amazon.com/exec/obidos/tg/detail/-/B000004286/

Any additional help to get me on the right path is appreciated. Thanks fro the help!

Here's the script:

function Get_String_Between_Strings ($contents, $starttag = ">", $endtag = "<", $number = FALSE) {

$matches = array ();

$starttag_length = strlen ($starttag);

//$contents = strtolower ($contents);

if (strstr ($contents, $starttag)) {

	while (strstr ($contents, $endtag)) {

		$contents = strstr ($contents, $starttag);
		$contents = substr ($contents, $starttag_length);

		$tostrip = strstr ($contents, $endtag);

		$matches[] = str_replace ($tostrip, "", $contents);

	}

}

if ($number) {

	$slice = array_slice ($matches, 0, $number);

	return ($slice);

} else {

	return ($matches);

}

}

superwormy

And what does teh function return?

And no, you shouldn't need to edit ANYTHING in the actual function itself.

One min I gotta go try it on that page...

superwormy

Seems to work OK, returns:

Main Title Listen Listen
Gift of a Thistle Listen Listen
Wallacer Courts Murron Listen Listen
Secret Wedding Listen Listen
Attack on Murron Listen Listen
Revenge
Murrons Burial
Making Plans/Gathering the Clans
Sons of Scotland
Battle of Stirling
For the Love of a Princess
Falkirk
Betrayal & Desolation
Mornays Dream
Legend Spreads
Princess Pleads for Wallaces Life
Freedom/The Execution/Bannockburn
End Credits

that part of the page.... if you don't set the limit var ( the last one ) it returns two other occurences as well.

superwormy

You realize that the page you're actually refering to is a redirect right?

Are you sure you're actually get the pages contents even?

brianb

Try it out...let me know what it returns for you...

I'm getting:

Array()

And that's it...

brianb

Hmmm...Ok...

Lemme try again...

How exactly did you pull it off?

Also, the page isn't a redirect (I don't think) I think it's just adding session elements to the end of the URL once it's accessed.

Trying this now...

superwormy

You can't just add elements to the end of a URL after a person requests it... it's definitally a redirect.

Check to make sure that you actually have the contents of the page in the variable that you are calling the function with...

I literally copied and pasted the source of the page it redirects to into a string, escaped teh characters, and copy pasted my function with your call to it into a page.

If you still have problems, email me your entire script and I"ll take a look and see if I can find the problem with teh function / script.