Using strpos to extract a link... Need Help

demo

I have been trying to find a way to extract all the links on a site and place them on my site. The only problem though is that whenever I try to search for a site, it tells me that there are no links on the site. Here is the code so far.

function search() {
global $config,$url;
$filename = "$url";
$handle = fopen($filename, "r");
$read = fread($handle, 100000);
$content = explode("\"", $read);
$result = strpos($content, "http://");
if ($result === false) {
echo "We could not find any links on this page. Please try again!";
} else {
echo "$result ";
}
}

Weedpacket

 $content = explode("\"", $read);

So $content is an array. strpos() expects it to be a string, so converts it into the string "Array". And the string "http://" does not appear in the string "Array".

What's the explode() for?

TheDefender

I assume you expect the links to be delimited by the backslash ( \ )character? Guessing that's why you are exploding?

If that's the case, you need to loop through each of the values returned in the $content array:

function search() { 
global $config,$url; 
$filename = "$url"; 
$handle = fopen($filename, "r"); 
$read = fread($handle, 100000); 
$content = explode("\"", $read);

for($i = 0; $i < count($content); $i++){
  $result = array();
  $result[] = strpos($content[$i], "http://"); 
}

if (empty($result)) { 
  echo "<font size=\"2\" face=\"verdana\">We could not find any links on this page. Please try again!</font>"; 
} else { 
  FOREACH($result AS $v){
    echo "<font size=\"2\" face=\"verdana\"><b>$v</b><br>"; 
}
}

This may need some tweaking. I wrote it quickly as I was finishing up my cup of coffee and heading out the door.

demo

Thanks for the help dude.

TheDefender

No problem... Let me know how that works out for you.

demo

I just tried this out on my site and it still does not seem to work. When I try to find a site the result returned is just blank.

here is the site

http://www.phpmini.com/demo/crawler

demo

I dont mean to dubble post but I still get the error whenevery I try to find a link on a site.

drew010

im not sure strpos is the easiest way to go. you can make a loop to capture when [url]http://[/url] starts and then find the terminating character to the link, but it is a lot of work. this code works to capture everything that begins with [url]http://[/url] on a page. the only problem is that it matches images as well that are referenced via http in <img> tags. you can probably make a workaround for it or decide how you want to match http. its really easy if you dont want to extract links that are typed on a page and you only want to grab hyperlinks, then change the pattern to
"/<a href=.http:\/\/...../"
here is the code that worked for me.

<?php

$url = "http://www.yahoo.com";

$data = file_get_contents($url);

if (preg_match_all("/http:\/\/[^\"'\s]+/", $data, $matches)) {
  print_r($matches);
} else {
  echo "No links on page.";
}


?>

demo

I got that to work but then how do I take away the array part. Here is what the result looks like.

Array ( [0] => Array ( [0] => http://www.gamingvoid.com/home [1] => http://www.gamingvoid.com/home/ ) )

I want it to look like

http://www.gamingvoid.com
http://www.gamingvoid.com/home

drew010

just iterate over the array with a simple foreach or for loop. i just posted that as an example, not a complete solution.

demo

would this work?

function search() {
global $config,$url;
$filename = "$url";
$data = file_get_contents($filename);
if (preg_match_all("/http:\/\/[^\"']+/", $data, $matches)) {

foreach ($matches AS $t) {
echo $t;
}
}
}

Sorry if I dont understand it that well. i did not know about preg_match_all untill today.

drew010

yeah its close. use
foreach($matches[0] as $t)

also make sure you get the \s in the pattern, that matches a whitespace which helps the script know that a url cannot have whitespace in it so it should stop capturing the url at the space.

preg_match_all("/http:\/\/[^\"\s']+/", $data, $matches)

preg is pretty nice, it really can be the solution to many problems like this.

demo

If I wanted to crawl even deeper into a site as in search other links on a site how would I do that?

drew010

you would probably need to use slightly more advanced code so you can also determine relative links, like <a href="/folder/page.html"> which implies the link resides on the current server and then you just file_get_contents that page. add all the urls you have spidered into an array of pages you've visited so you dont do them twice. also write code to compare [url]http://[/url] links to the current domain you are on so you know if they are linking using their entire site address.
it'll take much more code to get it right and write a fully functional (mostly) error-free spider.

http://sourceforge.net/projects/phpcrawl/