Web Page Scraper in PHP?

jely

I've been looking all over for a PHP script that I can use to to go a sports site, scrape the standings and re-display them on my site (preferably in my format as well). Going in to manually update is becoming a real pain.

So far I've had NO luck finding a PHP script that actually returns anything -- I've tried starting simple, just returning the text between the [title] tags, but the best I've gotten from any script I've found is a blank return page...no errors, but no information returned either.

Anyone know a simple way to do this in PHP? My preference is to not write the scraped standings to a database, but if that's what I gotta do, I'm game.

Any help is appreciated - thanks!

-= j =-

swr

First get the page into a variable. Like this:

$webpage = '';
$f = fopen('http://www.example.com/page.html', 'r');
while (!feof($f)) {
    $webpage .= fread($f, 1024);
}
fclose($f);

Then use a regular expression to extract the info:

preg_match('|<start><tags>(.*?)</end></tags>|is',
 $webpage, $matches);
echo $matches[1];

Weedpacket

There was a post on the very same subject a couple of days or so ago; someone wanted to copy sports results from another page and display them on his site in his format. Then he read the site's terms of usage agreement and found that he was forbidden from doing so without explicit written agreement from the publisher. Otherwise legal proceedings may ensue.

jely

1st, thanks for the legal warning -- I'll be sure that any site I scrape from doesn't disallow it -- seems pretty public domain, but I'll make sure to check first.

onto the script itself...I used the code snippet posted (thanks swr), and it's up here - but it returns a "timeout" on line 13:

$webpage .= fread($f, 1024);

any ideas? right now, jus to get it working, I set it it to try and return the "title" text from my site's homepage...but no dice.

thanks.

s_shrum

Originally posted by jely
...seems pretty public domain...

Just like mp3's in kazaa! :-S

Sorry, I had to say it.

jely

Anyone know why I'm getting that "timeout" on the line I mentioned? I tried increasing the memory buffer to a huge number and that didn't do it.

Oh, and I can't see the connection from copyrighted creative materials and factual statistics of sports standings...but like I said, if this ever gets working I'll be sure to do it on the up-and-up.

thanks.

swr

It's probably blocking trying to read the page.

What is the URL that you are fetching? Does it work when you bring it up in a web browser?

jely

swr -- yes, the page comes up fine:

http://www.leafsinsider.com/

(right now I'm still thinking "first things first" and just trying to establish a connection and return the text in the [title] tag - once I have something working I'll look at the more complex job of grabbing the standings I eventually want to use this for)

any advice?

swr

Works for me. Using the following code doesn't print anything (the preg_match fails for obvious reasons) but it's not timing out at all.

$webpage = '';
$f = fopen('http://www.leafsinsider.com/', 'r');
while (!feof($f)) {
    $webpage .= fread($f, 1024);
}
fclose($f);
preg_match('|<start><tags>(.*?)</end></tags>|is',
 $webpage, $matches);
echo $matches[1];

Maybe the site was down when you tried it? That can be a major problem when building a site that depends on another site; the other site can go down and there's nothing you can do about it.

There should probably be an "or die('Could not fetch page');" after the fopen, in case you can't make the call due to safe mode restrictions or something.

jely

Originally posted by swr
Maybe the site was down when you tried it?[/B]

nope -- could the issue be that I'm trying to set up the scraper on the same domain that I'm currently trying to scrape (to get it working)?

http://www.leafsinsider.com/scraper.php

here's the code I did (changing the tags to the title tags):


$webpage = ''; 
$f = fopen('http://www.leafsinsider.com/', 'r'); 
while (!feof($f)) { 
    $webpage .= fread($f, 1024); 
} 
fclose($f); 
preg_match('|<title>(.*?)</title>|is', 
 $webpage, $matches); 
echo $matches[1];

???

swr

The symptoms are consistent with your server being unable to figure out where www.leafsinsider.com is (even though it is on itself). This is usually the result of the server being misconfigured or the DNS servers it uses for lookups being down.

You can't go by IP to bypass the name lookup either, because the server uses virtual hosting (multiple domains on the same IP).

I don't think this is anything we can help you with. You'll have to contact your hosting provider.

jely

I tried changing the URL to scrape to CNN, Yahoo, and Google...and still can't get a title to return from any of them.

So swr, are you saying that my host will not allow me to scrape? I thought your comments meant my site itself could not be scraped, but now I'm not so sure.

thanks.

EDIT: well, moved the script to another server I do work on and it works like a charm, so it's my crappy host. time to change hosts!

thanks everyone.

swr

If your server's name service is completely misconfigured, it won't be able to look up any names.

Try http://216.239.53.99/ which is www.google.com. If that works, but http://www.google.com doesn't, then your hosting provider's name service is broken.

jely

crappy godaddy plan seems to be the culprit -- I tried the IP for Google and no dice there either.

Big thanks swr -- the PHP you gave works, just not on the server I want to use it on 🙂

Weedpacket

If memory serves, I remember reading something to the effect that Google blocks attempts at this sort of thing.

jely

well I was able to extract the "title" text from google in a test, so they may block search results from doing this.

I've seen attempts to scrape news headlines, too...but not sure if that works on google.