Parse search results from external site

alexu_

hi all,

I am trying to get the search results from an external site (let's say site.com), parse it using regular expressions and insert all records into my database. I am using the http navigator class found on php classes and I am sure this class works because I used it many times.

The problem is this: site.com creates a session when I first visit the site and this session will stick with me for all the pages I visit. This session is storing 3 cookies on my computer:

[ASPSESSIONIDSCSSDBDR] => Array ( [secure] => [value] => KFMKOJDDOIJNENCDLKOAPABD [date_added] => 1107452909 ) [ASPSESSIONIDQCRTCACR] => Array ( [secure] => [value] => PIPKFKDDMAFCGGOEENILPPAO [date_added] => 1107452909 ) [ASPSESSIONIDQARSDACQ] => Array ( [secure] => [value] => KJDLEKDDLHMNOKFALGKFBFFF [date_added] => 1107452912 )

So, in my script I visit the homepage and the search form page to get the cookies and then I try to get the search results page by sending the cookies and posting the form data.

All I get is this:

Warning: Page has Expired

The page you requested was created using information you submitted in a form. This page is no longer available. As a security precaution, Internet Explorer does not automatically resubmit your information for you.

But if I visit the page normally and then get the cookies from my cookie directory in Mozilla and insert them manually into the script it works. (of course it works for 30 minutes until the session expires)

so, because the session expires and I will have to run the parser daily, I have to get a new session every time and use it.

any help or hint (even tiny one) will be highly appreciated because this is the hardest problem I ever had in my 4-years PHP career 😕

thanks

rehfeld

it looks like they are using sessions, and invalidate them after a certain amount of time. so your session ids are expired.

you could first try and dynamically modify the [date_added] value to a more recent timestamp.

if that doesnt work, you will need to make your script first request the page that assigns those cookies, grab all thier values, then proceed to the results page.

alexu_

that's what I'm doing:

visit the homepage and the search page first to get all the cookies from the response headers, and then go to the search results page sending all the cookies and post variables.

the homepage sets two cookies and the search form page sets the third.

and the dates are exactly the time of the script run because every time I run the script it creates a new session and sets new cookies.

alexu_

I didn't find a solution yet....

Anyone???

rehfeld

some other things i would try:
use ini_set to set php http_user_agent to that of a real browser. they may be checking for that header and flagging you cause of the lack of it.

id also check the html source of thier pages. maybe they are using an image or javascript to try and see if its really a browser. like for example
<img src="image.asp">

a normal browser, would request that image.asp file and send the session cookie info to it. they could be invalidating your session serverside if your fail to request that image or whatever. if they really wanted to be slick they could even use an image.gif and have thier webserver execute an asp page when that file is requested, to futher hide what they are doing.

jus throwing out ideas...if i had a popular webservice that i didnt want bots to use, those might be some of the things i would do to try and prevent them

you know you could actually check pretty easily. try disabling images and javascript in your browser and then see if you can still use the service.

alexu_

I did the following:
1. visited the site using Mozilla
2. checked the 3 cookies into the Mozilla settings
3. got their values
4. manually set cookies values in my script (to the ones I got from Mozilla)

and it worked. The page displayed without any problem.

So I think it's a problem related exclusively to the cookies, or how are cookies set.

Anyway, the class authenticates itself as a Mozilla browser, I disabled the images and the site worked fine, but javascript cannot be disabled since they post using a javascript function (to check the form).