Detect crawlers ?

MichaelHe

Is there a way to detect if it's a search engine crawler visiting a site ?

What I was looking for is a way to have a text line only to be shown when it's a "normal" hit. And if it's a crawler this line should not be shown.

jkurrle

Here is a technical document that lays out a few possible web robot tests that you might use to determine if the hit you are getting is a web robot.

Good luck!

MichaelHe

Sound a bit complicated ?
When a web statistic shows how many hits a site has from google bots, yahoo bots and so on, does it really detect mouse movement etc. to get this info ?

MichaelHe

What about cookies ?
Does web crawlers set cookies ?

I have a cookie set on the site I use to count new visiters.
When a visiter visits the site for the first time it sets a cookie and add 1 to a database. The next time they visit the site check if there is a cookie, and if it's already there the visiter is not count again.

Can I use this cookie maybe.
After SET Cookie check if it's there, and if it's not there it could be a crawler ?
I don't think many "normal" users have set cookies to disable.