Indexing PHP pages

nobru

Hi all,
I already posted kind of the same question, but could not get any answer... so if anybody has a clue... 🙂

A few questions:

does a crawler/spider generate a session ID when browsing a site?
If yes, is it its "own" session ID, eg a session ID it generated itself?
Can it index this session ID?
On my index page, I have a piece of PHP code saying:
if(user_logged_in( ) = true){
display_this...
} else {
display_this...
}
Is there a risk that it is not indexed properly?
Eg is there a risk that the crawler indexes the supposed-to-be-for-logged-in-people part?

Thanks,
Nobru

Weedpacket

Depends on how the spider is written. It would certainly be feasible to write a spider that keeps session variables, but I don't know of any that bother (most spiders don't index pages that look dynamically-generated at all, on the assumption that the pages' contents would be changing too frequently for the index to remain current.)

But what I want to know is: how is a spider going to log in in the first place?

nobru

Hi Weedpacket,
I have very little knowledge about how the spiders and co. work...

I would like to make sure that a spider cannot "steal" the session ID of another user (or worse of an administrator) by browsing the site at the same time as the user (or admin).

Seems to me that this should not happen, but 'd like to get a confirmation.

superwormy

You are misunderstanding both spiders and sessions here...

Spiders are simply another user-agent, a different kind of browser ( just like Internet Explorer, Netscape, etc. ) that is browsing your website systematically.

Anything that a normal internet user, a normal user-agent like IE or Netscape can do, a Spider can do as well. Anything a normal user-agent CANNOT do, a Spider CANNOT do. They are the same damn thing.

For sessions, the user-agent DOES NOT EVER generate a session ID, or at least it shouldnt' be doing this. You are generating the session IDs and sending them to the user-agent, which is simply propogating the session ID everytime they visit a page.

If you're using cookies to propagate session IDs, the spider will most likely NOT propagate the ID, and most likely your app will assign a new ID to it everytime it visits a page. Spiders do not ( or at least not that I've seen, although I'm sure you could build one that did so if youw anted ) login to anything.

If you're propagating the session IDs through URLs, the spider most likely will not follow your URLs, because spiders tend to avoid following URLs with query strings, for several reasons. One, theres no way to tell that tehse two pages are the same page:

page.php?SESSIONID=52
page.php?SESSIONID=64

And two, because dynamically generated pages tend to change a lot, too much to keep the index updated, as someone pointed out above.

It is possible for ANY user-agent ( not just a spider, but Internet Explorer, Netscape, etc. ) to steal another users session ID. That's why its important to have time-outs on session IDs, and important that they are random. The only time this would happen is if the random number generated was the same as someon eelses, very unlikely if you're doing it correctly.

nobru

Thanks a lot for your answer superwormy, much clearer to me now 🙂

"It is possible for ANY user-agent ( not just a spider, but Internet Explorer, Netscape, etc. ) to steal another users session ID."

This is another discussion, but do you have any idea about how to do that and prevent it, except time-outs?

"and important that they are random"
But they always are, aren't they? I thought they were based on microtime or somthing...

superwormy

There is no way to prevent this entirely... but to start...

Make sure your sessions time-out after a certain amount of time. Theres really no reason to keep a session more than a day in most instances, what are the chances a person is goingto stay on your website for 24 full hours?

You CANNOT check by IP address, this is what lots of people at first think they should do and ITS A MISTAKE. LOTS of people ( specifically AOL users, AOL users very wierd proxying techniques to speed up browsing and ease load ) have IP addresses that can change EVERYTIME THEY VISIT ANOTHER PAGE. So don't use the IP address.

You could check by the HTTP_USER_AGENT server variable, becuase really a person's user-agent shouldn't be changing while they are browsing your site. Record that user-agent in relation to each sessionID, and if you get a sessionID where the stored and current user-agents don't match, give them a new ID.

superwormy

Oh, it also depends on how you are generating session IDs, I've seen peopel generate them as auto-incrementing numbers before... hehe which is definitally not random :-)

This page has some good examples of what to use for unique session IDs:
http://www.php.net/manual/en/function.session-id.php

nobru

About the IP check, I have the following function. I didn't check it out yet, but thought this could make the deal for AOL users for instance:

if(getenv('HTTP_CLIENT_IP')) {
$ip = getenv('HTTP_CLIENT_IP');

} elseif(getenv('HTTP_X_FORWARDED_FOR')) { //Most proxies add the HTTP_X_FORWARDED thing
$ip = getenv('HTTP_X_FORWARDED_FOR');

} elseif(getenv('REMOTE_ADDR')) {
$ip = getenv('REMOTE_ADDR');
} elseif($SERVER['REMOTE_ADDR']) {
$ip = $SERVER['REMOTE_ADDR'];
} else {
$ip = "undefined";
}

Anyway, what I don't get is that I use this to check that a user is logged in, not to prevent people from stealing a session. How can you relate a session to a user-agent or even ip-address? It supposes that your session data is stored into a db, doesn't it?

About generating the session id, why bother? Isn't the 'default' one good enough?

Weedpacket

Originally posted by nobru
About generating the session id, why bother? Isn't the 'default' one good enough?

I'm generally happy with it myself: it's basically an MD5 of the microtime; an internal pseudorandom number generator; and if you specify an external source of entropy in php.ini, it will also pull bits from that and shovel them in.