session/ user/ spider question

craigh

(I'm working with Postnuke here, but someone with good PHP & session knowledge can hopefully answer this question)

I need to know what happens when a spider 'hits' my web site - specifically (and technically) how sessions are involved. Do spiders 'load' a page as a normal user Agent would?

When a 'normal' user (guest or registered) comes to my site, they are (1) assigned a session id (by PHP or apache?) and they are (2) entered into the *_session_info table and they (3) load the page they requested (lets say the front page), triggering any code that may be on that page.

is it possible that a spider can do both #1 and #2, but not #3 - therefore not triggering code on the pages?

craigh

bump - PLEASE!

Shrike

Spiders act as standard users - look in your server log for googlebot 😉

Your server can issue a session id for the spider, but I'm not sure if spiders will retain cookies and whatever. Any code that triggers for a standard user will trigger for the spider too, they just do standard HTTP requests.

If you don't want spiders doing this, create a robots.txt file in the root of your web, most spiders will follow the directives put in there.

craigh

Originally posted by Shrike
Spiders act as standard users ... Any code that triggers for a standard user will trigger for the spider too, they just do standard HTTP requests.

Is there anyway for spiders to visit a site and NOT trigger the code on a standard page?

Shrike

Ask Google about robots.txt

daveyboy

if you don't want a bot to get a session, using cookies to store the session id is the best way. bots behave much like a normal user with a couple of exceptions: they usually obey robots.txt; they generally won't download image files or other multimedia; they ignore javascript; some don't like frames; they ignore cookies. there are probably other things, too. that's just a general rule, and many bots are different (spammers' bots ignore robots.txt, image search engine crawlers don't ignore images).

if you actually want to bypass code if the browser is a bot, you'll probably have to use $_SERVER['USER_AGENT'] and compare it with the names of bots.

craigh

to clarify:
I'm NOT trying to STOP bots or spiders. I don't need to change robots.txt or find if a hit is a spider or not (yet).

what I am trying to understand is HOW THEY WORK. The issue is that I have some code on a page that, for some reason, some users run and some do not. I am trying to figure out if a spider could possibly 'hit' a page and not run this code.

I know that some users are not running this code because it basically writes their session id to a log and there is evidence that users exist without being logged. I know this because we have rewritten the session handlers to record the session id in a table instead of in cookies. So ... users are appearing in the session table and not in my log. Make sense?

Any help is appreciated!

Shrike

You can't write the session id to a table instead of a cookie. The only two ways to propogate sessions are through cookies or as a URL parameter. I guess you mean you use a database rather than file based sessions by setting session_set_save_handler();

If a user has cookies disabled then your session tracking won't work (unless you also use the URL method also)

craigh

Well, part of my problem is my lack of knowledge of sessions - that' part of what I am trying to solve 😉

I am using Postnuke and the code I am trying to run occurs in a block on each page. Postnuke has rewritten all the session handlers and there is a *_session_info table that has a session id, first time logged in, most recent access, etc.

I'm seeing entries in that table, but nothing in my log, so I am trying to figure out why.

Any help?

daveyboy

what are you expecting to see in your log? i think phpnuke uses cookies to store its session id. they wouldn't show up in the web server logs.