IP Ranges

NZ_Kiwis

I want to issue a token for users on my website which will be linked to a sessionID, this will be saved in a database and used later.

What I want to avoid is tokens being issued to robots which crawl my website, how can I do this?

I think via a text file I can remove robots but I don't want to do this, just not issue a token.

I thought if there was an IP range for Google, Yahoo, Bing etc I could exclude them but I'm sure there are hundreds of robots out there?

Any ideas?

bradgrafelman

NZ_Kiwis;11029639 wrote:
What I want to avoid is tokens being issued to robots which crawl my website, how can I do this?

One idea is to generate that token using a hidden image (e.g. just something that makes the users' browser send a request for that URL). You could then use a robots.txt file to tell (responsible) robots they aren't allowed to crawl that URL.

NZ_Kiwis;11029639 wrote:
I thought if there was an IP range for Google, Yahoo, Bing etc I could exclude them but I'm sure there are hundreds of robots out there?

Probably. Probably even more than that. And such a range probably isn't published for general public knowledge. And even if it was, it probably has changed over time anyway. (In summary: don't use IP ranges. 🙂)