Determine if its Google or Yahoo spidering my site, or is it actually visitors

[bradgrafelman]

And what happens when Google buys another IP range? Or Yahoo? Or MSN? Why check for several ranges that Google might use when you could instead do a simple string match for "Googlebot" in the user agent?

shrug To answer your question, the client's IP address is stored in $_SERVER['REMOTE_ADDR']; more information about this superglobal (and the others) can be found here: [man]variables.predefined[/man].

[woocha]

That makes really solid sense...I like that idea and will try going with that. Thank you. When I run into coding snags, I will post here again and hopfuly I can get somemore help

[Paul_Ferrie]

This thread should be stickied. Extremly useful. 🙂

[woocha]

Yeah I agree...A lot of webmasters might find this sort of info on spiders and bot very useful

[Paul_Ferrie]

I have been storing unique user agents on my forum today. I know that at least 2 visitors were spider bots but they dont sure in my database.

I typical entry in the agent coloum is

Mozilla/5.0 (Windows; U; Win 9x 4.90; de; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11

Can someone tell me what i should see if it is a google, yahoo bot etc.

Thanks

[woocha]

You might consider checking if the ip is listed on iplists.com or do a reverse lookup on dnsstuff.com

[Paul_Ferrie]

I use network-tools on my forum. Though my database does not have an entry for this ip.

i think i will give it another day of gathering unique hits then have a look over then see if any bots have been added.
That iplist link will come in very handy thanks

[alanzhao]

I use

            gethostbyaddr($ip)

usually you will get result like: 123.123.122.222.googlebot.com or 123.123.123.123.somecrawler.whatever.net

[Pehape]

Woocha, Are u still checking the IP range instead of bot string ?

[woocha]

Pehape wrote:

Woocha, Are u still checking the IP range instead of bot string ?

Can you please show a way to check for bot string, and maybe where to find bot strings for major search engines?

[Paul_Ferrie]

woocha wrote:

Can you please show a way to check for bot string, and maybe where to find bot strings for major search engines?

Post number 10 on the previuos page :rolleyes: 😃

[digital_skwerl]

I'm dealing with a similar issue but am a bit ahead of woocha. Hope this helps...

I've collected a list of user agents from http://www.user-agents.org/

I created a table 'bots' to store these strings, and add on to the list as I identify new bots.

I then simply compare the user agent string (gathered with $_SERVER['HTTP_USER_AGENT']😉 to the records in my bots table. If there are no matches, than I can continue with my stat.

Here's some code...

$agent = $_SERVER['HTTP_USER_AGENT'];

$query_bot = "SELECT bot_id FROM bots WHERE user_agent = '$agent'";
$result_bot = @mysql_query($query_bot);
$check_bot = @mysql_num_rows($result_bot);
if($check_bot == 0) {

// Your Query Here
} 

[bretticus]

I have seen some huge discrepancies between stat trackers. Keep in mind, the google code is javascript based and thus, search engine bots will not execute that code, whereas a serverside tracker will track every ip address. Also, keep in mind that you may define "visit" differently than google. You may be tracking every page hit, for example, and they may be taking each individual remote host and looking for inactivity of say, 30 minutes, to define a visit. Also, some ISP users will have multiple ips, due to proxy servers. Here are some helpful scripts to get unique ips.

Using those same scripts here with a snippet of code from one of my old projects, I'll demonstrate a simple way to detect a search bot. I have no idea if the bot names are the same. This is old code (excluding the functions) and may need some refinement.

<?php
$is_bot = false;
$search_bots = array("msnbot","Googlebot","LocalcomBot","IRLbot","Ask Jeeves","Yahoo! Slurp","Gigabot");
foreach ($search_bots as $bot) {
	if ( strpos(getip(), $bot) !== false ) {
		$is_bot = true;
		break;
	}
}

// ... do your magic ...

function validip($ip) {
    if (!empty($ip) && ip2long($ip)!=-1) {
        $reserved_ips = array (
        array('0.0.0.0','2.255.255.255'),
        array('10.0.0.0','10.255.255.255'),
        array('127.0.0.0','127.255.255.255'),
        array('169.254.0.0','169.254.255.255'),
        array('172.16.0.0','172.31.255.255'),
        array('192.0.2.0','192.0.2.255'),
        array('192.168.0.0','192.168.255.255'),
        array('255.255.255.0','255.255.255.255')
        );

    foreach ($reserved_ips as $r) {
        $min = ip2long($r[0]);
        $max = ip2long($r[1]);
        if ((ip2long($ip) >= $min) && (ip2long($ip) <= $max)) return false;
    }
    return true;
} else {
    return false;
}
 }

 function getip() {
    if (validip($_SERVER["HTTP_CLIENT_IP"])) {
        return $_SERVER["HTTP_CLIENT_IP"];
    }
    foreach (explode(",",$_SERVER["HTTP_X_FORWARDED_FOR"]) as $ip) {
        if (validip(trim($ip))) {
            return $ip;
        }
    }
    if (validip($_SERVER["HTTP_X_FORWARDED"])) {
        return $_SERVER["HTTP_X_FORWARDED"];
    } elseif (validip($_SERVER["HTTP_FORWARDED_FOR"])) {
        return $_SERVER["HTTP_FORWARDED_FOR"];
    } elseif (validip($_SERVER["HTTP_FORWARDED"])) {
        return $_SERVER["HTTP_FORWARDED"];
    } elseif (validip($_SERVER["HTTP_X_FORWARDED"])) {
        return $_SERVER["HTTP_X_FORWARDED"];
    } else {
        return $_SERVER["REMOTE_ADDR"];
    }
 }
?>