Spider? Crawler? Dataminer? Help!!

gjdunga

I need to create a script that will load a webpage, Submit to the page a cookie to validate it's allowed, and then pull several fields from a table and store the data in mysql.

It will be ran as a cron job to update a message board. Basicly we are allowing users into a message board, based on there activity on another webpage..

No, I don't have access to the webpages backend. 🙁

I do have access to the content I need, via a cookie, and can use excel to do it manualy, but want to automate the task.

Considering my php skills are not uber, i figured i'd ask for some brainstorming here. I've thought about a dataminer, or a spider, but very few deal with submitting a cookie back to authenticate the page request.

Also the page takes over 30 seconds to load, as it has about 9000+ records this one page displays.

Once i yank the data, it will be shoved into a mysql database with the exact same field names, as the page displays.

The cron job will empty the datbase out everynight and start new, as the webpage is the "real" container.

Any suggestions? Please Nail me!
gjdunga //aim / yahoo / hotmail (IM)

Weedpacket

You'll probably want to use fsockopen() to open the connection through which you make the request and retrieve the response. That way you've got full control over the headers you send, including the "Cookie:".

Read the file in in chunks (the [man]fsockopen[/man] manual page still providing an example, although that on [man]fread[/man] can alsobe looked at, as that's the name of the function you'd use.) Since it's such a large page that takes more than 30 seconds, put inside that loop a line that calls [man]set_time_limit[/man] to a reasonable delay to wait for one chunk. Then, if the other server dies or something at least your page won't be waiting forever (which is what would happen if you set_time_limit(0) outside the loop).

gjdunga

I think I understand..
Remember you have a NEWBIE here..

if (!($request=fsockopen('whatever.com',80,$errno,$errstr))) exit($errstr));
else {
  socket_set_timeout($request,10);
  fwrite($request,"GET /index.html HTTP/1.1\r\n\r\n");
  do $header.=fread($request,1);
  while (!preg_match('/\r\n\r\n$/',$header));
  if (preg_match('/(?i)transfer\-coding:\s+chunked\r\n/',$header))
   do {
     unset($byte);
     unset($chunk_size);
     do $byte=fread($request,1);
     while (preg_match('/[^a-zA-Z0-9]/',$byte));
     if (!($chunk_size=fread($request,4))) break;
     $chunk_size=hexdec($byte.$chunk_size);
     $response.=fread($request,$chunk_size);
   } while ($chunk_size);
  else {
   $status=socket_get_status($request);
   $response=fread($request,$status['unread_bytes']);
  }
  fclose($request);
}
exit($response);

(Shamelessly stolen from the FSOCKOPEN Thread)

I know i need to submit the headers (reading up as we speak)
to that the cookie value's will be submitted, and that the contents will be dumped into a flatfile or array.