Avoiding test site being cached

amy_damnit

If I get an account with some ISP, is there a way to put up my test website but avoid it being cached by sites like Google?

There are lots of things for which I want/need a more production environment (e.g. having access to an e-mail server), but I do not want my test website and code forever ending up on Google or The WayBack Machine.

Any ideas?

Also, do I have to have a domain name in order to set up a test environment with an ISP?

Amy

NogDog

You can use a robots.txt file in the domain's home directory to ask robots (such as Google's) to ignore certain files and/or directories.

As far as the test domain goes, if you have an existing domain where you can work, you could simply create a sub-domain there (e.g. "test.yourdomain.com"), have it point to a new sub-directory under the main domain, then put your test files under that directory as though it were the web document root directory for the site being tested.

amy_damnit

NogDog;10921446 wrote:
You can use a robots.txt file in the domain's home directory to ask robots (such as Google's) to ignore certain files and/or directories.

How effective are those, though?

And to clarify, technically the moment I put anything up on the Internet, someone is crawling and/or caching it, right?

As far as the test domain goes, if you have an existing domain where you can work, you could simply create a sub-domain there (e.g. "test.yourdomain.com"), have it point to a new sub-directory under the main domain, then put your test files under that directory as though it were the web document root directory for the site being tested.

Yes, I know that, but what I was still wondering if you even need a domain?

I have an account with some big ISP a few years ago, and I think I just accessed my site like www.SomeBigISP.com/YourAccount.html or whatever.

And this point also relates back to the first questions.

If I have any kind of legitimate domain, I don't want Google or The Wayback machine or whoever to eternally link my test website with "www.MyReallyImportantDomainName.com" if you follow?!

Anonymity is important to me until - and if ever - I would go live.

Amy

NogDog

The robots.txt file will be honored by Google and other major, legitimate players. It is in no way binding to any robot that feels like ignoring it, however.

In any case, though, nothing can be crawled by any such robot unless there is a link to it from some already crawled page that the robot knows about. So as long as you do not provide a link to any of your test pages anywhere on the web where a robot could "see" those links, those pages will never be crawled. (I.e.: the robots simply follow links they discover in a page, the follow any links in those new pages, and so forth. If there is no link to those test pages that the robots can discover, then they will never find them.)

amy_damnit

NogDog;10921454 wrote:
The robots.txt file will be honored by Google and other major, legitimate players. It is in no way binding to any robot that feels like ignoring it, however.

In any case, though, nothing can be crawled by any such robot unless there is a link to it from some already crawled page that the robot knows about.

So I basically have 100% anonymity by "security through obscurity" until my website link starts showing up on the Internet?

[qoute]So as long as you do not provide a link to any of your test pages anywhere on the web where a robot could "see" those links, those pages will never be crawled.[/quote]

So that is really the only way your webpage/website would appear on Google or The Wayback machine, is if it was linked to on other websites?

I suppose if someone Googled it, that would be another way, right?

(I.e.: the robots simply follow links they discover in a page, the follow any links in those new pages, and so forth.

Ad that is where they got their names, right?

Amy

NogDog

Yep, spider-bots just follow links. If they never find a link to a page, then they will be totally ignorant of its existence. Only "gotcha" I can think of would be if the files are in a directory, that directory is linked to somewhere on the web ("the web" includes other publicly accessible pages on your site), and there is no default file in that directory and your web server is configured to display a directory listing when a directory is requested that has no default file. (I always disable that feature on my sites as a potential security hole.)

NogDog

amy.damnit;10921457 wrote:
...
I suppose if someone Googled it, that would be another way, right?
...

Google only knows about a page if its spider-bot has found it and stored it in their database. However, should someone for some reason guess a URL, they could access it. You could, however, add HTTP authentication to that directory, requiring anyone (including robots) who try to access any file in that directory (or sub-directories) to log in. (I often do that if putting a test site on the web for testing or, more often, so that the client can evaluate it without it being publicly accessible.)

amy_damnit

NogDog;10921467 wrote:
Yep, spider-bots just follow links. If they never find a link to a page, then they will be totally ignorant of its existence.

Interesting. So the only way to get "know" is if you are already "know"?!

Google only knows about a page if its spider-bot has found it and stored it in their database.

Again, so just "being" on the Internet is not enough to get you listed and/or cache.

However, should someone for some reason guess a URL, they could access it. You could, however, add HTTP authentication to that directory, requiring anyone (including robots) who try to access any file in that directory (or sub-directories) to log in. (I often do that if putting a test site on the web for testing or, more often, so that the client can evaluate it without it being publicly accessible.)

Funny you should mention that, because that is what I was thinking of doing before I posted my question.

That way - as long as I'm using a dummy domain name - if they do "find" me, then all they would find is the log-in page, right?

I mean Google can't cache or index a page that requires you to be logged in, right?

Amy

Weedpacket

amy.damnit wrote:
Interesting. So the only way to get "know" is if you are already "know"?!

Well, if you don't want to wait for others to start linking to you, you can explicitly tell Google about your site. Their rating system does take how often it is referred to elsewhere, but ultimately of course it's all about the content.

I mean Google can't cache or index a page that requires you to be logged in, right?

Even if it did process forms it wouldn't know what to put in the form fields....

DaveRandom

{AHEM}.... Errr?? Authentication??

DaveRandom

Plus, search engines wont know of the existence of your site unless they find a link to it from somewhere else... especially if you dont assign any kind of DNS records to your IP. Also, putting your site on a non-standard port (I use :54345 a lot) is a good way to stop malicious spiders etc from finding it.

amy_damnit

DaveRandom;10921692 wrote:
{AHEM}.... Errr?? Authentication??

Hi. Not sure what that means??

Plus, search engines wont know of the existence of your site unless they find a link to it from somewhere else... especially if you dont assign any kind of DNS records to your IP.

So how do you NOT assign any kind of DNS records to your IP?

(To me, that implies not having a domain and just using the IP that your web-host gives you.)

Also, putting your site on a non-standard port (I use :54345 a lot) is a good way to stop malicious spiders etc from finding it.

How do you do that when using a web-host?

Amy

NogDog

amy.damnit;10921760 wrote:
Hi. Not sure what that means??

HTTP Authentication, as I mentioned in Post #7, and you replied that you were already thinking about that.

amy_damnit

NogDog;10921788 wrote:
HTTP Authentication, as I mentioned in Post #7, and you replied that you were already thinking about that.

No, Dave random said:

Originally Posted by DaveRandom View Post
{AHEM}.... Errr?? Authentication??

And I replied...

Hi. Not sure what that means??

I know what you and I were talking about.

Amy

NogDog

Yes, and I'm suggesting that he was talking about the same thing: HTTP Authentication.

DaveRandom

Tis true.

If a client is not authenticated they can't see the page at all, so the search engines etc can't possibly index the page. As far as I am aware that would be the only way to 100% guarantee it...

CoderDan

For any dev site I wanted to guarantee was never seen by anyone else, I'd use .htaccess for basic authentication. It's simple, works most places, and keeps my php code clean of premature auth routines.
You can get a hosting account without a domain name, and they will give you a way to access it from their server (something like http://www.hosts.com/~user/www/) or directly via IP address (ie: http://192.168.0.1/)
The latter is better if you ever use absolute pathing in your code.
As for using a different port, free web hosting will never(?) give you the ability to set up or modify apache config, which is where that would need to be done, although most places that you have an actual host at will, and even come with cpanel to allow you to set up your virtual hosts in.
To be perfectly honest, however, you're really best off to just set up some wamp package (assuming you have windows) on your own machine, and work on it right there.

amy_damnit

CoderDan;10921980 wrote:
For any dev site I wanted to guarantee was never seen by anyone else, I'd use .htaccess for basic authentication. It's simple, works most places, and keeps my php code clean of premature auth routines.

I knew about that from locking access, but I just wasn't sure if by me accessing .htaccess restricted areas on my site if it would still end up on the internet because of Google or whoever caching it.

From earlier posts, it doesn't sound like that would be a problem.

You can get a hosting account without a domain name, and they will give you a way to access it from their server (something like http://www.hosts.com/~user/www/) or directly via IP address (ie: http://192.168.0.1/)
The latter is better if you ever use absolute pathing in your code.

Yah, that is what I was saying earlier.

It seems like that is one way to make sure your domain is not associated with test files.

To be perfectly honest, however, you're really best off to just set up some wamp package (assuming you have windows) on your own machine, and work on it right there.

Well, I am doing that now, but eventually you need to upload to a web host to test. More so, there is certain functionality that I can't do from home like sending out PHP e-mails. (I asked about that a long time ago on here and the responses I got were - use a web host.)

What do you think is a compromise on these issues?

Amy

CoderDan

amy.damnit;10921990 wrote:
I knew about that from locking access, but I just wasn't sure if by me accessing .htaccess restricted areas on my site if it would still end up on the internet because of Google or whoever caching it.

From earlier posts, it doesn't sound like that would be a problem.

Yep. no self respecting will index a page that returns a 401 Authorization Required

amy.damnit;10921990 wrote:
Yah, that is what I was saying earlier.

It seems like that is one way to make sure your domain is not associated with test files.

That's for sure.

amy.damnit;10921990 wrote:
Well, I am doing that now, but eventually you need to upload to a web host to test. More so, there is certain functionality that I can't do from home like sending out PHP e-mails. (I asked about that a long time ago on here and the responses I got were - use a web host.)

What do you think is a compromise on these issues?

Amy

Well, I'm not sure about mail on windows, haven't done it in a while. I ran a linux box as a test server myself, both with and without domain names, and had it sending emails fine, Ant from memory, it wasn't that much trickier to get windows to send emails, was just the sendmail path.

But yeah, for the top anonymity on a live test server, don't use a domain name (unless it's one you set up yourself in windows/system32/drivers/etc/hosts) and put some htaccess auth on it, and you're golden.
Also, if all you want to test live is emails, then just put your mailing script live with a wrapper to test it, put it in an odd folder with an odd filename, and write basic getstring/session authenetication into it and test it standalone. All you need to test mail for is if you're sending attachments and/or html anyway, and just to make sure it looks ok in the inbox. otherwise you can replace your mail() call with a function to dump it into a text file or db, and be done with it for testing. mail either sends or it doesn't, and if it doesn't it's a server setup issue 99% of the time, not a code issue.

Weedpacket

amy.damnit wrote:
From earlier posts, it doesn't sound like that would be a problem.

Yup. In the end web crawlers aren't magic - they can't do anything any other http client can't do and servers treat them in exactly the same way they treat browsers - send back responses to requests that they send.