sending 404 to bots

sneakyimp

when my site encounters an error, i want to send a 404 to the client if it happens to be a bot.

i have the bot detection code working...is this how i send 404?

<?php
header("HTTP/1.0 404 Not Found");
?>

also, i know it's hard to know for certain, but will search engines be likely to stop crawling those pages if i return the 404 like that?

ClarkF1

Why not use a robots.txt file???

More info here: http://www.searchtools.com/robots/robots-txt.html

MarkR

Search engines don't necessarily assume that a "404" error (or any other type of error) is permanent.

An intelligent spider will probably schedule any not found URL for another attempt later, in case it's a temporary problem (e.g. site has been closed for maintenance, replaced by a holding page).

Those pages won't be removed from the index either, unless the spider gets 404s repeatedly over several attempts. At some point, the page will be de-listed. Eventually, the spider will stop trying to fetch that page, and remove it from its list.

Not all spiders are that clever though. The ones I have the most problems with are evil / lame ones, which don't usually identify themselves as robots anyway. Google, MSN, Yahoo, etc, don't usually give problems.

Mark

Weedpacket

MarkR wrote:
Search engines don't necessarily assume that a "404" error (or any other type of error) is permanent.

That's what a "410" is for. But then, a bot that's smart enough to know what "410" means is likely to be one of the sensible ones anyway.

But neither 404 nor 410 sound appropriate here. Sounds more like a 403.

sneakyimp

i have a robots.txt file but to add ALL these URLs would be quite painful.

the basic problem is that my site has some pages that call up different data depending on the query string provided and i made the mistake of assuming that ALL of the possible query string arguments were valid for every page when i created my google sitemap. so the googlebot is crawling invalid pages in my site and filling up my error log with junk.

these pages were never valid and never will be.

thanks weedpacket for the advice about other error codes.

i'm still wondering if the PHP i posted is the correct way to send a 403/404/410 code?

bradgrafelman

I would probably add an exit or die() as well.

sneakyimp

10.4.4 403 Forbidden

The server understood the request, but is refusing to fulfill it.
Authorization will not help and the request SHOULD NOT be repeated.
If the request method was not HEAD and the server wishes to make
public why the request has not been fulfilled, it SHOULD describe the
reason for the refusal in the entity. If the server does not wish to
make this information available to the client, the status code 404
(Not Found) can be used instead.

Hm. Sort of implies that I should 'make public why the request has not been fulfilled'. How would one do that?

Weedpacket

The same way you'd make a 404 response for people who stumble onto one; a (presumably simple) page in the request body that in this case would say something to the effect that "This page is for viewing by sentient beings only".