.htaccess and bot help

msnhockey

Got a msg from my host saying that im using excessive queries on one of my databases and told me to check the stats for unwanted spammers and such

I have looked at the stats page and found a bot that is using excessive resources but i have to idea on how to block this bot accordingly

Unknown robot (identified by 'bot*') (HITS) 566588+3485

its not a very busy site just for some friends and i
here is the stats page if this helps
http://msnhockey.ipage.com/stats/index.html

Thanks in advance to who can help

Im guessing i probably need to use robots.txt

Regards,
Mike

bradgrafelman

Not sure what it is we're supposed to be helping you with... what does "identified by 'bot*'" mean? What data was being analyzed when that identification was made? User agent string? DNS name? My grandmother's favorite color?

msnhockey

I dont know what it means either and there aint any other info beside it to get the DNS, Ip address

on my stats page link that i included that is what it showed
Unknown robot (identified by 'bot*')

I dont know if I should block all bots and allow the good ones or find a way to block certain ones. All i know is something accessing my site is forcing my database queries to exceed 50,000 per hour which is against my host TOS and that Unknown robot is the one accessing my site the most.

sneakyimp

I am not sure at all what your log looks like -- the link you sent requires login -- but I suspec the 'identified by bot*' bit refers to the user agent of the remote system accessing yours.

In PHP, you can check the user agent of a remote user and just [man]die[/man] based on what you find there. This is not especially powerful because the remote user can report whatever they want in this value because it is (optionally) supplied with each page request.

$user_agent = isset($_SERVER['HTTP_USER_AGENT']) : $_SERVER['HTTP_USER_AGENT'] ? "";
if (strpos($user_agent, 'bot*')) {
  die("Sorry, access is not permitted");
}

The remote address can also be used to filter your remote users. phpBB and other open source projects usually let you maintain a ban list of ip addresses (or ip ranges) that let you deny access to all kinds of remote users. How you implement it is up to you:

$remote_address = isset($_SERVER['REMOTE_ADDR']) : $_SERVER['REMOTE_ADDR'] : "";
if ($remote_address == '11.22.33.44') {
  // note that banning a single ip address this way is not very powerful.
  die("Sorry, access not permitted");
}

sneakyimp

By the way, why is this in client-side technologies?

johanafm

Go with robots.txt, and either set it to deny / (if you don't care about turning up in search engines, which you probably won't if it's just "for some friends") or specify deny for any scripts that accesses the db.