Detecting Bots

[mlat]

Okay, i'm making a pretty sweet content suite for about 5 months. It is going VERY well, and im nearing completion.

One of the things that has been a bother, is bot usage. I want bots to be able to come on, I think its great, but they're giving me trouble.

First of all, all my session handling is coded manually. I never liked PHPs built in session functions, so I made my own, they work GREAT. There are two kinds of sessions, cookie sessions and URL sessions. Basically, if my script cant store session id in a cookie, it passes it through a URL, much how forums work.

Here's the stinker: Whenever bots come online (I found out google was accessing my beta test site), they create new sessions for every page load they do (I have no clue why they do not properly use my code for sessions). Thus, it appears that I have like 30 guests online instead of 1 bot session. I need two things: The most basic thing I need, is a quick, easy, secure way to tell the difference between a bot and a user. I already have all my session code finished, it just needs compatibility for bots added. Secondly, a method of identifying bots would be nice as well. I know bots have some sort of ID, but I wouldn't know how to get it. If I had it, I'd compare it in the database with something and then I'd have a way of identifying a bot.

Many thanks in advance.

[MarkR]

Determining whether something is a bot is not easy. A lot of bots don't announce themselves as such. I've seen bots claiming to be all sorts of normal web browsers.

I'd recommend that you create sessions as lazily as possible - i.e. only create them when they hit a page which requires them to have one.

Then ban robots from pages which require sessions, using robots.txt

I've also had the problem of robots hitting my non-robot-compatible pages, even though they're not allowed to by robots.txt

I've knocked up a piece of code to check whether something appears to be a bot, from its user-agent:

<?php
/*
 * Ban robots from pages we don't want them in. This file
 * should only be require()'d in pages we don't want bots in.
 *
 * For maximum performance, require this before anything else.
 * it has no dependencies.
 */

function ForbidRobots()
{
	$ua = @ $_SERVER['HTTP_USER_AGENT'];
	# No user agent is fine.
	if (! $ua) { return; }

$botre = '/
# this is an extended regular expression. whitespace is ignored
# unless escaped, and comments like this are allowed.

(bot\b)| # words ending in "bot"
(\bbot)| # words starting with "bot"
(spider)|(spyder)|(crawl)| # parts of words common in robot UAs
(bookmark\ search)| 

(url\ control)|(libwww) # Libraries commonly used in bots
(http\ ole\ control)|
(wininet)|
(winhttp)|
(\blwp)| # some libwwwperl user agents start with this
(httplib)|


# Specific bots etc. Note that we dont need anything called "bot"
# as that will be caught by other rules above.
(bunnyslippers)|
(cherrypick)|
(download)| # a lot of bots are leechers or downloaders
(check\b)|(checker\b)| # link checkers etc
(offline\ explorer)|
(frontpage)|
(getright)|
(httrack)|
(harvest)|
(webzip)|
(webstrip)|(webreap)|
(web[\ \.]image)|
(web\ sucker)|
(webcop)|
(teleport)|
(webwhack)|
(wget)|
(\bzeus)|
(archive)|
(frontpage)| # no thanks
(moget)|(webfetch)|
(leech)| # No legit UA ever contains the word "leech"
(test\ user\ agent) # For testing only
/ix'; // i = case insensitive, x = extended.
	if (preg_match($botre, $ua)) {
		// It's a bad bot. Get out as soon as possible. 
		// Provide a helpful message, in case a human sees it
		header('HTTP/1.0 403 Robots forbidden');
		header('Content-type:text/plain');
		echo("Robots are not allowed to access this page\n");
		echo("If you believe you're seeing this message in error, " .
			"contact the site owner. You should be able to find their contact details ".
			"from the site homepage.\n");
		exit;		
	}
}


ForbidRobots();
?>

Mark

[mlat]

Thats the issue, I can't ignore em or block them. They need to be on there indexing stuff.

[MarkR]

mlat wrote:

Thats the issue, I can't ignore em or block them. They need to be on there indexing stuff.

But they don't need sessions.

Don't start a session unless the user does something which is relevant to a session, e.g. log on or register. Block bots from your logon and register pages with robots.txt.

Then robots (and indeed, other unregistered users) should not clutter up your sessions.

Mark

[mlat]

Okay, I'm making a content suite for other people to download and/or buy. Guests are included in certain statistics and such, and they have information in their session of what they are viewing so people can see the online list and what everybody is doing.

Bots are good, as they index things for people who would use my script into search engines. I know its possible to detect them properly.

I have just figured out how I am going to identify most of them. All I need help with now is a way of preventing them from making new sessions after they've made just one.

[mlat]

Okay guys, I got it working. Bots are now properly using sessions, and some of them are being identified by my script.

Just a tip, censorship is very rarely a good answer for anything. If you block out the bots, then you block out potential search engine listings.

[MarkR]

We don't use the nobots.php script in our pages that are intended for use by bots.

The only pages which include that code are ones which bots should never access.

We don't store any persistent data on people who don't either logon, register or add an item to their basket. Robots are not allowed to do any of those things (in robots.txt), so I'm just further enforcing my robots.txt policy by banning robots which ignore it.

Mark

[mlat]

Hmm, okay, I can see what you need them for then.

Thanks for your help 🙂