URL VAlidator !

capte

heres what i wanna do :

i wanna get a list of url i've got in my data base and test the urls to see if there valid ... I've found this script but it only does it for one url at a time ... Anyone could help on this ?

heres the working script !

function checklink($link)
{
//see if we have a link to check
if($link)
{
//open the url
$file = @fopen ("$link", "r");
}
if($file)
{ return true ;
fclose($file);
}
}
if($link !="")
{
//show result if good
if((checklink($link))=="true")
{
echo "<center><b>Good Link 🙂</b><br>($link)</center>";
}
else
{
//show result if bad
echo "<center><b>Bad Link 🙁</b><br>($link)</center>" ;
}
}
?>
</head>
<body>
<form action="index.php" method="post">
<table border="0" align="center">
<tr><td>
<font size="1" face="verdana">
Input Link e.g (http://blah.com)
</font>
</td>
</tr>
<tr>
<td>
<textarea name="link" cols="30"></textarea>
</td></tr>
<tr><td>
<input type="submit" value="Check Link">  <input type="reset" value="Reset">
</td></tr>
</table>
</font>
</body>
</html>

devinemke

Originally posted by capte
i wanna get a list of url i've got in my data base and test the urls to see if there valid ... I've found this script but it only does it for one url at a time

so then loop thru the list and run the link checker during each iteration.

stutterbug

Further to the previous post, try this:

<?

function checklink($link)
{
  //see if we have a link to check
  if($link)
    {
      //open the url
      $file = @fopen ("$link", "r");
    }
  if($file)
    { return true ;
    fclose($file);
    }
}

function printstatus($link) {
  print $link;
  if($link !="")
    {
      //show result if good
      if((checklink($link))=="true")
        {
          echo "<center><b>Good Link </b><br>($link)</center>";
        }
      else
        {
          //show result if bad
          echo "<center><b>Bad Link </b><br>($link)</center>" ;
        }
    }
}
$linkarray = split("\n",$HTTP_POST_VARS['link']);
foreach ($linkarray as $linkitem) {
  printstatus($linkitem);
}
?>
</head>
<body>
<form action="test.php" method="post">
<table border="0" align="center">
<tr><td>
<font size="1" face="verdana">
Input Link e.g ([url]http://blah.com[/url])
</font>
</td>
</tr>
<tr>
<td>
<textarea name="link" cols="30"></textarea>
</td></tr>
<tr><td>
<input type="submit" value="Check Link">  <input type="reset" value="Reset">
</td></tr>
</table>
</font>
</body>

Here, I am assuming that you are inserting multiple URLs, each on a new line (if you are using commas, replace the '\n' with a comma, etc.)

Hope it helps.

Sco_________

your php will probably time out before it finishes unless you have your timeout set high in the php.ini. I do the same sorta thing, but after

printstatus($linkitem); i'd put flush(); for a neat affect 🙂.

capte

Stutterbug and Scott ! ... thanks a lot ... not exactly working ... but i guess we're quite close !

here's my code again ... with my db feeding the info in the box ...
Only trouble here is that i get a lot of "BAD LINK" when there actually good !

so what do you think ?

function checklink($link)
{
//see if we have a link to check
if($link)
{
//open the url
$file = @fopen ("$link", "r");
}
if($file)
{ return true ;
fclose($file);
}
}

function printstatus($link) {
print $link;
if($link !="")
{
//show result if good
if((checklink($link))=="true")
{
echo "<center><b>Good Link </b><br>($link)</center>";
}
else
{
//show result if bad
echo "<center><b>Bad Link </b><br>($link)</center>" ;
}
}
}
$linkarray = split("\n",$HTTP_POST_VARS['link']);
foreach ($linkarray as $linkitem) {
printstatus($linkitem);
flush();
}
?>
</head>
<body>
<form action="index.php" method="post">
<table border="0" align="center">
<tr><td>
<font size="1" face="verdana">
Input Link e.g (<a href="http://blah.com" target="_blank">http://blah.com</a>)
</font>
</td>
</tr>
<tr>
<td>
<textarea name="link" cols="30">
<?
mysql_connect("localhost","*","**");

mysql_select_db("****");

if(!isset($cmd))
{

$result = mysql_query("select adresse FROM repSites");
$count = count($result);
//run the while loop that grabs all the news scripts
while($r=mysql_fetch_array($result))
{
//grab the title and the ID of the news
$adresse=$r["adresse"];//take out the title

 //make the title a link = <a href='index.php?cmd=delete&id=$id'>Delete</a>
  echo $adresse;
  echo "\n";
}

}

?>
</textarea>
</td></tr>
<tr><td>
<input type="submit" value="Check Link"> <input type="reset" value="Reset">
</td></tr>
</table>
</font> </form>
</body>

Capte ! 😕

stutterbug

The problem is, I am pretty sure, many major websites drop connections made by non-standard clients (like the bot you created) as a matter of security -- giving you a lot of false negatives (I notice that cnn.com fails, for example). A LOT of these sites have plenty of experience with scripted attacks and poorly written, bandwidth-consuming, home-built spiders so they are screwed down pretty tight.

It surprises me that PHP doesn't have some sort of programmable web client interface like Perl does, but I have never heard of one and I can't see one in any of the documentation. Maybe someone here can help. I am pretty sure you could solve the problem by telling PHP to report itself as a Mozilla or Internet Explorer user agent though, again, I don't know if it is possible and I have not seen any such method documented.

I would say that you you may be better off using a pre-written Perl script (not as much fun, I realize) to check your links. The Perl module I have used in the past for client emulation is LWP. The library's home site is http://lwp.linpro.no/lwp/ and it has sample scripts (including some link checkers) there. This library is pretty powerful and will allow you, for example, to fill out forms (if you know what the forms should contain) and respond to dynamically generated server challenges. Serious muscle for a simple scripting language.

Please post back on what you find.

Weedpacket

Originally posted by stutterbug
It surprises me that PHP doesn't have some sort of programmable web client interface like Perl does, but I have never heard of one and I can't see one in any of the documentation.

Well, there's [man]cURL[/man] support built in; there are also http client packages in PEAR. phpclasses.org also has a bunch of http client libraries, and classes like apache_lookup_uri which can be used when PHP's own [man]apache_lookup_uri[/man] function is not available. (Notice that of those five links, only two go to a non-php.net site).

Please post back on what you find.

Well I found this by searching these forums for "LWP". Does that count?