The spider won't talk back.

nfzgrld

Here's one to chew on. I have a search engine script that includes a real live spider. It's a good spider, as it parses and stores all the things I want it to. It understands robots, and many other things. The only major problem I have with it is I can't seem to find a way to get it to talk. I would like it very much if it could tell a server when it's been crawling around so people could look at their stats and know it's been there, but I am at a loss. How do I get the thing to send a string to the server it's grabbing pages from?

Nick:rolleyes:

sarahk

You might find what you need at one of these sites:

The information you add is in the User Agent info - but I'm not sure how you manipulate that. Botspotter gets it from $HTTP_SERVER_VARS['HTTP_USER_AGENT']. Feel free to hit one of my sites to test - a few times (not hundreds).

Effectively you want to give either an email address or a web address which explains why you have visited. Try and get a unique name. Test it in google with a search for Name robots.txt and see if anyone else is there.

Weedpacket

First of all, the server needs to be listening. The SMTP protocol requires that all mail servers have a "webmaster@example.com" address, but a lot of sites blackhole it.

nfzgrld

SMTP? I'm not sure how that is relevent, but hey, what do I know? Using the HTTP_USER_AGENT is not useful becasue it's a server level setting. There may be other scripts on a server using that for one reason or another, so I can't have it set to something specific for this script. If I can do an ini_set on it that might be just the thing for this, but I'm not sure that's possible. We'll know soon as i just tried it and we'll see in a minute what happened as I did hit your real estate link a grand total of 1 times. That should be enough for the thing to show up if it worked. Look for "Axel Search" in your logs.

Nick

sarahk

I've checked out the logs for PCPM but no sightings?!

If the user agent is set right you should see your visit through the public part of Botspotter too.

nfzgrld

Yeah, I was going about it the wrong way. Actually, I was pretty close to begin with, but I missed something.

I was doing this:
ini_set(user_agent, $spiderhost);

That was actually very close. I went through a ton of stuff, then I realized something and did this:
ini_set(user_agent, "$spiderhost");

It turns out that the user_agent in php.ini is remmed out by default, so when this wasn't working, nothing showed up. When I unremmed it, and left it set to "PHP" it showed up. Then when I used ini_set, it did what I told it.

BANG! Works everytime. When I get a chance I'll hit your site again so you can see the results. Won't be till much later though, I have to take my daughter to the orthodonti$t. 😃

Nick

Weedpacket

Oh, I was just wondering how the spider was going to tell the server it visited so that people could know to look at their stats. My understanding was that they wouldn't know it had visited until they had looked at the stats and saw that the spider had left its User-Agent string behind, but how would they know when to look at their stats?

sarahk

Hi Weedpacket

You're right, people would need to know to look at their stats and would have to be interested enough to follow up on it.

At best he will get some new hits to his site.

At worst, from my opinion, he'll be building his site's PR by leaving behind the info. Alot of stats packages give links to the referrer which is often manipulated to be a fake site, this builds PR which is then passed to the real, usually adult, site. Read more at: http://www.weberblog.com/article.php?story=20031121220439862

Still more sites give links to frequent referrers which again builds PR. Not that PR might be the goal. Traffic probably is. But if people don't know about the site then this is a good way to spread the word.

Sarah

nfzgrld

Originally posted by Weedpacket
Oh, I was just wondering how the spider was going to tell the server it visited so that people could know to look at their stats.

Well, yes, they have to go look at their stats. If they don't already know to do that regularly, there isn't much I can do for them. When I said "talk" I meant report the user agent to the server, not actually talk to the site. I'm not even sure that's possible. 🆒

Weedpacket

Originally posted by nfzgrld
When I said "talk" I meant report the user agent to the server, not actually talk to the site.

Right, I misconstrued. My bad.

nfzgrld

Originally posted by sarahk
At worst, from my opinion, he'll be building his site's PR by leaving behind the info. Alot of stats packages give links to the referrer which is often manipulated to be a fake site, this builds PR which is then passed to the real, usually adult, site. Read more at: http://www.weberblog.com/article.php?story=20031121220439862

Still more sites give links to frequent referrers which again builds PR. Not that PR might be the goal. Traffic probably is. But if people don't know about the site then this is a good way to spread the word.

Sarah [/B]

Yeah, I've caught a couple of these. You can use robots.txt to stop the spider from hitting your stats, which you should have done anyway, but the best thing to do is just put an htaccess on it, or better yet, set up security in the httpd.conf if you have access to it.

sarahk

You can use robots.txt to stop the spider from hitting your stats, which you should have done anyway, but the best thing to do is just put an htaccess on it, or better yet, set up security in the httpd.conf if you have access to it.

Those options are technically possible, however I disagree that they are the best thing to do. Chasing their spiders or IP addresses would become a full time job. I know from building spam filters how clever and manipulative these people can be. I'd visited several of their spam sites before I realised what was going on.

Better by far to protect your stats, make that page non-spiderable through robots.txt (reputable spiders use it, and we're only worried about reputable search engines). Better still to protect the stats pages from public access.

Sarah