link harvesting from google

[Scubanaked]

HI my script opens files, urls and even files with arguments but it can't seem to open the google search file, is there some way to capture the response to a post and store it in a way I can manipulate it.

So far:
<?php
$webfile = fopen("$http://www.google.com/search?q=naked&hl=en&lr=&ie=UTF-8&start=90&sa=N", "r")
while(!feof($webfile))
{
$fileline = fread($webfile, 1024)
//And here I manipulate thedata
}
fclose($webfile);
?>
Thanks Scubanaked : )

[Angus]

I was trying to do something similar - I wanted to query a domain in Google and return the number of links that existed to a site so I could analyse link popularity between multiple sites, but for some reason Google spits out an error.

My same script works with MSN, Excite, AlltheWeb, Yahoo and more, but Google is being a pain 🙂

Had a mate who is a bit of a guru look at the problem and he couldnt understand why it wouldnt return the correct result either...

I ended up using Yahoo - as Google is the fall through result. Not as accurate - but better than nothing.

[ednark]

when you make a query for a webpage with fopen... php makes a http request and identifies the browser making the request as... wouldn't you know... PHP. Google checks the user agent and as PHP isnot as a real broweser.... throws an error

so i bet you that gooles servers are set to not respond to browser who says they are a script....

either open your own socket, and write you own request part by part... so you can fake your user_agent...

or and even worse solution... do a call to lynx... and read in the source from what lynx returns...

[Scubanaked]

Geez I never thought to try yahoo. That will take care of my need. Thanks
Brian
Check out my Scuba sIte and the fourteen pound lobster I caught two weeks ago at www.scubanaked.com

: ):p :p