Hiding Pages from Bots, Engines, Etc.

ffdave

I have numerous pages that require user names and passwords in order to gain access to them. I would like these pages hidden so a search bot does not archive them and when a user searches, he/she can't see the protected content in results.

How do I go about hiding this, i read about meta tags but is there anyway else I can protect against these bots? My web hosting company uses Apache, is there any way to add a configuration file in the directory to hide pages/folders?

frikikip

check http://www.thesitewizard.com/archive/robotstxt.shtml.

You can create a robots.txt file. This file can contain a set of rules for search engines, site rippers, etc allowing or denying certain actions.

Some site rippers however, don't use the robot.txt to check what they are allowed to do. In that case you might want to use Apache's "Mod Rewrite" which is a lot more powerfull.

For more info on ModRewrite check: http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html.

Greets,

Marvin.

Weedpacket

You can also use a .htaccess file to control access to different directories, etc.; the server will use that to decide whether requests should be satisfied or not.

frikikip wrote:
Some site rippers however, don't use the robot.txt to check what they are allowed to do.

Of course, a site ripper (or other spider, or anyone who doesn't have a valid username and password) wouldn't have a username or password, so they wouldn't see the protected content anyway - that's the whole point of having a username and password in the first place, after all.