Determine if its Google or Yahoo spidering my site, or is it actually visitors

woocha

Hey guys.

I had a guys write this code for me a while back before I started to learn PHP. The script is a stats tracker. But, it must have some problems. To see if it was working, I started using Google Analytics as well. The Google stats tracker says I get 100 visits a day and the home brew version says I have 1000. I am tending to believe the Google stats gizmo, but I want to fix the original one that was made for me. SO here is my question:

Is it possible to determine that it is Google's or Yahoo's Spider checking out my site.
I'm thinking something like if(Google or Yahoo){don't count};

Is there anyway to do this?

Thanks fellas and Happy New Year !!

MarkR

Not guaranteed; OEM stats sites usually work by putting some javascript on your site- this only counts browsers which support Javascript and choose to allow it in that context.

Robots don't normally understand Javascript at all, but moreover, many bots are "evil" bots which don't respect either the robots exclusion protocol, or identify themselves as such.

You can match the robot's user-agent for that of known robots, but other than that there's little you can do. "evil" bots will usually claim they are a web browser and come from arbitrary (rather than well-known) IP addresses.

Googlebot and other "good" robots are well documented in their user-agent string and originating IP.

Mark

woocha

OK...so smaller or mean bots/spiders aside, you think I should look for google's spider IP address and not count it when It visits...is that correct?

bradgrafelman

You can prevent most of the decent spiders by examining the User-agent header. Google for some common ones.... Google even has this page to help you block Googlebot.

woocha

OK...here's a question. Is it possible to track how many times Google or MSN or Yahoo or Alta Vista spiders my site? I think this could also be useful information.

elibizif

You could install BBClone 0.49b on your site. I have it running on all three of mine. It keeps track of robots/spiders in red and regular visitors in blue.

woocha

Yeah I could, but I want to make my original script do this, not add another counter to the mix....Any other thoughts?

mikell

This from google:
The IP addresses used by Googlebot change from time to time. The best way to identify accesses by Googlebot is to use the user-agent (Googlebot).

So if you want to ID a google spider/bot, look at the user-agent. If it contains the
string "Googlebot", its most likely google (or another bot pretending to be google which, I would guess, is very unlikely). The $_SERVER variable, which you're probably familiar with, contains the user-agent information you need.

woocha

mikell wrote:
This from google:
The IP addresses used by Googlebot change from time to time. The best way to identify accesses by Googlebot is to use the user-agent (Googlebot).

So if you want to ID a google spider/bot, look at the user-agent. If it contains the
string "Googlebot", its most likely google (or another bot pretending to be google which, I would guess, is very unlikely). The $_SERVER variable, which you're probably familiar with, contains the user-agent information you need.

Thank you for the tip, but what about the other bot, like those from Yahoo or MSN or AOL?

bradgrafelman

Yahoo! Slurp
MSNbot

List of Known Robot User-Agent Fields
List of User-Agents (Spiders, Robots, Browser)
etc.

woocha

I read on another forum that I might want to think about segregating by IP address. I was directed to a website that has something like 300 IP's on file for these spiders. Unfortunately, I have no idea how to code a simple script to read an IP and add it to a counter....Is the IP logged somehow through the $_server variable?

bradgrafelman

And what happens when Google buys another IP range? Or Yahoo? Or MSN? Why check for several ranges that Google might use when you could instead do a simple string match for "Googlebot" in the user agent?

shrug To answer your question, the client's IP address is stored in $_SERVER['REMOTE_ADDR']; more information about this superglobal (and the others) can be found here: [man]variables.predefined[/man].

woocha

That makes really solid sense...I like that idea and will try going with that. Thank you. When I run into coding snags, I will post here again and hopfuly I can get somemore help

Paul_Ferrie

This thread should be stickied. Extremly useful. 🙂

woocha

Yeah I agree...A lot of webmasters might find this sort of info on spiders and bot very useful

Paul_Ferrie

I have been storing unique user agents on my forum today. I know that at least 2 visitors were spider bots but they dont sure in my database.

I typical entry in the agent coloum is

Mozilla/5.0 (Windows; U; Win 9x 4.90; de; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11

Can someone tell me what i should see if it is a google, yahoo bot etc.

Thanks

woocha

You might consider checking if the ip is listed on iplists.com or do a reverse lookup on dnsstuff.com

Paul_Ferrie

I use network-tools on my forum. Though my database does not have an entry for this ip.

i think i will give it another day of gathering unique hits then have a look over then see if any bots have been added.
That iplist link will come in very handy thanks

alanzhao

I use

            gethostbyaddr($ip)

usually you will get result like: 123.123.122.222.googlebot.com or 123.123.123.123.somecrawler.whatever.net

Pehape

Woocha, Are u still checking the IP range instead of bot string ?