How to handle this case?

zuihao

Hi!
I am making some kind project which handle search engine robot, scripted by PHP.
The script will only response the request from robot, by checking brower's HTTP_USER_AGENT.
But the question is this parameter can be faked.
Is there any good method to detect remote request is from a real robot? I know check IP is a good way, but as I know there are about 54000 computers of Google, I have no idea all of them IP, so this seem not good enough.

Thanks
Mike

blahpro

Well, the HTTP_USER_AGENT will not often be faked to look like google, so I see no reason why you could not implement this to get an accurate result.

Or, you could gethostbyip(); the IP, and use regular expressions to see if it contains the string "googlebot", or whatever google's bot's hostnames contain.

Dont hold me to it though! Its just a guess!

zuihao

Hi!
Thanks first!
I look through my web log, and find some robots have host name, but most don't have. So http_user_agent seems the only easy way.
Do you know any doc about how this parameter is generated?

Thanks
Mike

blahpro

Sorry, I'm afraid not 🙁

Weedpacket

Originally posted by zuihao
Do you know any doc about how this parameter is generated?

It's supplied by the client (in the "User-Agent:" header of requests it makes). The client can supply anything it likes.

zuihao

Hi!
The good news is I have made it by Delphi, it is easier than I though.
The bad news is to detect remote requester is a robot or not by this parameter is not a good idea.
:bemused:
I have to look for another solution.

Thanks
Mike