Blocking WebCopier

[cpegphp]

Hi guys,

is there a good way to block guys using webcopier or wget or others?

[[deleted]]

Not without going to great lengths.

A 'decent' webcopy script will announce itself as a webcopier, and will look for a 'robot.txt' (or something like that) to get instructions about what it is allowed to fetch.

But with things like teleport pro you can switch all the politeness off and just do raw GETs. The only way to find those is by checking the requests.

I suppose you could make a 'prepend' script that is executed whenever a page is loaded and which logs the request and remote-host.
If too many requests arrive from the same host, it's a copier and you can refuse the request.

However, all you have to do is set the politeness a bit higher on teleport pro and you can still download everything, it just takes longer.

[Anon]

You could make the user enter the site by clicking on a form submit button and only moving to the rest of the site if they do, so that just following links wouldn't work.

Or do all the navigation through java or javascript. Not very friendly but would probably work.

[Anon]

Another way is to make a "pitfall" link with the same fontcolor and backgroundcolor in your frontpage.
It should be unvisible for humans. But a dumb spider gets the link and you have caught it in your pitfall. 🙂
The script in the pitfall blocks the IP for many minutes in your other scripts or redirects him to some dirty site about copyright or so... 🙂

Regards
Ingo

[Anon]

You have to decide to let in Searchengines than Google, Fireball etc.

Identify their signature (Agent) and take them off your pitfall....

[[deleted]]

I love the pitfall idea, but how are you going to allow 'real' searchengines in if you have no clue about which servers are searchengines and which are naughty spiders?

And what if one of your users wants to save the page to disk using IE? or an offline copy? That will spider the page and save everything it finds... including the pitfall.

[gerardg]

robots.txt is one step.

I also use apache's mod_rewrite to block by user agent. Most website grabbers identify themselves by default, but this can usually be overridden and they (and scripts) can disguise themselves as web browsers. Then there's other ways to deal with that - I think mod_bandwidth might.

Look at the docs at apache.org for further info on using mod_rewrite. You can even find lists of user agents by searching the newsgroups.

[gerardg]

Here's something I found which can be used as a guide:

Hejsa!

Jeg har af en kunde fået til opgave at sikre, at man ikke kan "hotlinke" til
dennes content, og har på ApacheToday fundet følgende eksempel (skal
indsættes i en .htaccess fil i det bibliotek der skal beskyttes mod
hotlinking:

---

Options Includes FollowSymLinks
AddType text/x-server-parsed-html .html .htm .shtml .htm
AddType video/quicktime qt mov

AuthUserFile /dev/null
AuthGroupFile /dev/null
RewriteEngine on
RewriteOptions inherit
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^{http://.domain.com/.$} [NC]
RewriteCond %{HTTP_REFERER} !^{http://.domain.com:80/.$} [NC]
RewriteCond %{HTTP_REFERER} !^<a href="http://xxx.xxx.xx.xx/.$">http://xxx.xxx.xx.xx/.$</a> [NC]
RewriteCond %{HTTP_REFERER} !^<a href="http://xxx.xxx.xx.xx:80/.$">http://xxx.xxx.xx.xx:80/.$</a> [NC]
RewriteCond %{HTTP_REFERER} !^{http://localhost:8081/.$} [NC]
RewriteRule .[Jj][Pp][Gg]$|.*[Gg][Ii][Ff]$ http://localhost:8081/badref.htm

RewriteCond %{HTTP_USER_AGENT} ^DISCo\Pump. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip. [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber. [OR]
RewriteCond %{HTTP_USER_AGENT} ^{ExtractorPro.} [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE. [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet. [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Gets. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla. [OR]
RewriteCond %{HTTP_USER_AGENT} ^{Go-Ahead-Got-It.} [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula. [OR]
RewriteCond %{HTTP_USER_AGENT} ^IBrowse. [OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET. [OR]
RewriteCond %{HTTP_USER_AGENT} ^{Internet\Ninja.} [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar. [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView. [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\tool. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\PiX. [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite. [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider. [OR]
RewriteCond %{HTTP_USER_AGENT} ^{Offline\Explorer.} [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\Foto. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey. [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Slurp. [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison. [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport. [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto. [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier. [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch. [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper. [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger. [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper. [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker. [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP. [OR]
RewriteCond %{HTTP_USER_AGENT} ^{Web\Image\Collector.} [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\Sucker. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Webster. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget. [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch. [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver. [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp. [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut.
RewriteRule .[Jj][Pp][Gg]$ /leeches.html [L]

RewriteCond %{HTTP_REFERER} ^{http://.adfilter.com.$} [OR]
RewriteCond %{HTTP_REFERER} ^{http://207.198.147.$} [OR]
RewriteCond %{HTTP_REFERER} ^{http://www.neocities.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://chat.passagen.s.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://volpi.sti.com.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://batepapo0.uol.com.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://batepapo1.uol.com.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://batepapo2.uol.com.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://batepapo3.uol.com.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://batepapo4.uol.com.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://batepapo6.uol.com.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://batepapo7.uol.com.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://209.2.137.} [OR]
RewriteCond %{HTTP_REFERER} ^{http://207.126.121.$} [OR]
RewriteCond %{HTTP_REFERER} ^http://.bianca.com/.$ [OR]
RewriteCond %{HTTP_REFERER} ^http://.angelfire.com/.$ [OR]
RewriteCond %{HTTP_REFERER} ^http://.hotmail.$ [OR]
RewriteCond %{HTTP_REFERER} ^http://.yahoo.com.$ [OR]
RewriteCond %{HTTP_REFERER} ^http://.geocities.com.$ [OR]
RewriteCond %{HTTP_REFERER} ^{http://www.partyhouse.}$
RewriteRule .[Jj][Pp][Gg]*$ /leeches.html [L]

---

Det virker desværre ikke, er der nogen der har et forslag?

Serveren er forresten Windows 2000 Server med Apache 1.3.11

/jesper/

I have been using this method for a long time, I don't know how new that article is, but I used it a long time ago to not only block all the spambots I could find, but all of the software for mirroring my webpage as well.

Here is a longer list of common spam bots and mirror bots that I have been able to find:

SetEnvIfNoCase User-Agent "EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "CherryPickerSE" bad_bot
SetEnvIfNoCase User-Agent "CherryPickerElite" bad_bot
SetEnvIfNoCase User-Agent "Crescent" bad_bot
SetEnvIfNoCase User-Agent "EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "MCspider" bad_bot
SetEnvIfNoCase User-Agent "bew" bad_bot
SetEnvIfNoCase User-Agent "Deweb" bad_bot
SetEnvIfNoCase User-Agent "FEZhead" bad_bot
SetEnvIfNoCase User-Agent "Fetcher" bad_bot
SetEnvIfNoCase User-Agent "Getleft" bad_bot
SetEnvIfNoCase User-Agent "GetURL" bad_bot
SetEnvIfNoCase User-Agent "HTTrack" bad_bot
SetEnvIfNoCase User-Agent "IBM_Planetwide" bad_bot
SetEnvIfNoCase User-Agent "KWebGet" bad_bot
SetEnvIfNoCase User-Agent "Monster" bad_bot
SetEnvIfNoCase User-Agent "Mirror" bad_bot
SetEnvIfNoCase User-Agent "NetCarta" bad_bot
SetEnvIfNoCase User-Agent "OpaL" bad_bot
SetEnvIfNoCase User-Agent "PackRat" bad_bot
SetEnvIfNoCase User-Agent "pavuk" bad_bot
SetEnvIfNoCase User-Agent "PushSite" bad_bot
SetEnvIfNoCase User-Agent "Rsync" bad_bot
SetEnvIfNoCase User-Agent "Shai" bad_bot
SetEnvIfNoCase User-Agent "Spegla" bad_bot
SetEnvIfNoCase User-Agent "SpiderBot" bad_bot
SetEnvIfNoCase User-Agent "SuperBot" bad_bot
SetEnvIfNoCase User-Agent "tarspider" bad_bot
SetEnvIfNoCase User-Agent "Templeton" bad_bot
SetEnvIfNoCase User-Agent "WebCopy" bad_bot
SetEnvIfNoCase User-Agent "WebFetcher" bad_bot
SetEnvIfNoCase User-Agent "WebMiner" bad_bot
SetEnvIfNoCase User-Agent "webvac" bad_bot
SetEnvIfNoCase User-Agent "webwalk" bad_bot
SetEnvIfNoCase User-Agent "w3mir" bad_bot
SetEnvIfNoCase User-Agent "XGET" bad_bot
SetEnvIfNoCase User-Agent "Wget" bad_bot
SetEnvIfNoCase User-Agent "WebReaper" bad_bot
SetEnvIfNoCase User-Agent "WUMPUS" bad_bot
SetEnvIfNoCase User-Agent "FAST-WebCrawler" bad_bot