file_get_contents()

Alkarym

hi everyone.. i'm currently working on a "grabber" like program that read web pages and drop their content into a database...

everyting is working quite well but i'm having troubles with this site : http://www.kenmore.com/shc/s/home_10154_12604 i get an error while trying to open the page and read it! 😕

i've looked everywehre over the internet but found nothing to help me out so maybe one of you will find a way to read the source of that page! please help me out!

GeXus

What error do you get?

It is possible for sites to block file_get_contents... You may want to check out cURL.

Alkarym

i get the error "failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error in C:\wamp\www\Grabber\kenmore.php on line 26"

haven't tryed to check the url since i did not know it could "block" the file_get_contents()

i'm fairly new to php so please don't muck me >.< hehehe

GeXus

Try a different URL and see if it works.. how about yahoo.com?

Alkarym

i know that other url works since i made multiple grabbers in fact, since each site's code desing changes i have one grabber page for each company i need to get data from...

the code usually works fine and i can get the data, fill it in the DB and use it back. but i just can't go and read the table on that site... the URL i gave was the front page's URL but i can't grab anything from that site.

if you wanna try, you can always try to compare between www.geappliances.com and that first url in my first post and see the difference...

The "GE" site gives me a full string with all the page's source code. I then filter that code based on how the site is written to find the table with all the appliance's specs and insert the "relevant" data into my DB.

is there another way to "read" the html content of a page and sort out the tables data from that?

Alkarym

here's an example of an URL i would want to grab data from:

http://www.kenmore.com/shc/s/p_10154_12604_02246633000P?vName=Kenmore&cName=Cooktops%2C+Ranges+%26+Ovens&sName=Slide-In+Ranges

in the "specifications" tab you can see a table withh all the specs of the oven, normally, what i did on the other sites was to grab the source, read it to find the table with those specs and run a loop to gram the spec names and values and drop it in a DB using mysql.

as i said, up untill that site, i made 4 grabbers for 4 different companies and everything worked accordingly. this here site gives me an error and i'm stuck at that point!

GeXus


<?php

$ch = curl_init();

curl_setopt ($ch, CURLOPT_URL, "http://www.url.com/");
curl_setopt ($ch, CURLOPT_HEADER, 0);

curl_exec ($ch);

curl_close ($ch);
?>

If you have cURL installed, this can get remote pages as well.

chrome29

I have made some similar applications and have often come across sites that block "bots". I have been generally successful by adding the following line at the top of the script:

ini_set('user_agent',' 	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2');