[Resolved] Reading external file -> session/cookie absent problem

leatherback

Hi,

For work we have acces to this huge database of institutes, with over 10.000 institues in conservation / development aid. We would like to sent out a mailing to these institutes, so we need the adresses, to print our envelopes. And instead of copying each individual address by hand (Yes, each instute has it's own page, we cannot select, say, all institutes in one country) I was looking into writing a script which will grap the content of a page, looks for all the linked institute-pages, open each of those, and grabbing the address.

The problem I am now facing is that I get the "cookie is not set" error. When I normally open the site, I have registered acces through our institute IP, I suppose. But when I run the script (Using $lines = file($url)😉, I cannot acces the page. Anybody here an idea on how to go about it?

I have tried using javascript -> window.open and then var txt = wnd.document.body.outerHTML.value // var txt = wnd.document.body.innerHTML.value // var txt = wnd.document.body.documentElement.value but no succes there. I cannot read the content of the newly opened page. Probably a browser security thing?

Hope anybody here has some nice suggesteion on getting the addresses.

Thx!

AstroTeg

If you can, go with CURL and make sure you define the cookie jar. With the cookie jar defined, the site will be able to set the cookie (which, without looking, is probably a session) and let you in. If its not a session, but authentication, you could (with just a little bit more code) have CURL handle the authentication and keep the site happy.

If you really want to be smooth, Perl has some awesome robot/crawling libraries available to it. If this is something you might have to do more than once, it might be something to look into. If its a one time shot, then stay with PHP.