comprehensive bot detector script?

sneakyimp

i've searched this forum for script bits, but i was wondering if anyone has a reall good web crawler/bot detector script? i want to know if my pages are dealing with a search engine or a real user. this is the best i've found so far:

<?
$user_agent = $_SERVER['HTTP_USER_AGENT'];

function is_bot($user_agent) {
  $spiders = array('Googlebot', 'MSNBOT', 'FAST-WebCrawler', 'Gigabot', 'YahooSeeker', 'ZyBorg'); 
  foreach($spiders as $key=>$bot) {
    echo 'testing:' . $bot . '<br>';
    if (stristr($user_agent, $bot)) {
      return TRUE;
    }
  }
  // if we reach this point, we've tried all the bots with no match
  return FALSE;
}
echo 'result:' . is_bot($user_agent);
?>

this seems to work pretty well as far as i can tell--at least logic-wise. HOWEVER, my list of bots is pretty lame. Is there somewhere i can find a fairly comprehensive list of webcrawlers/bots? I've seen stuff like:

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

in my access logs and i have no idea if that's a bot or someone with a yahoo search toolbar.

justsomeone

There's a list here but it may not be utterly comprehensive.

This is a bit of a moving target, as spam bots, and some search engine crawlers don't always identify themselves as bots. Search engines don't like to think that you will show them something different than you would show to a real user. This isn't unreasonable - any attempt to do this is usally an attempt to "fool" the earch engines, and they don't like to be fooled.

It's a very risky strategy, as you can find yourself delisted by the search engines if they catch you at it. BMW's german site was recently bumped down to a pagerank of zero on Google. Read more here.

sneakyimp

Thanks for the tip. My intention is to leave off session id's for bots so that my site doesn't seem like an entirely different collection of URLs each time the bot visits because the session id differs. Apparently a lot of forum type applications have problems with bots relentlessly crawling them because of the ever-changing session ids. In other cases, the forum sites are dropped altogether.

frost110

Use cookies to pass session ids. If you do not have cookies enabled..well sorry you cannot view my site. Sucks for you. =)

--FrosT

sneakyimp

Um, with respect frost, that post isn't trying to answer my question now, is it?