[RESOLVED] PHP & RollingCurl

[kpowning]

I'm playing with the RollingCurl.php class that uses curl_multi for connecting to multiple remote sites, links, whatever. I have a couple of questions for efficiency.

1. Will curl timeout, return a response or what if lets say a website is overloaded and times-out? Can you perform a (Do While) loop then to make sure all the links actually successful as the connection to the remote site becomes available?

2. Does anyone have any examples of how I should handle this? or what I should be checking?

I have 50 pages on a site that I need to connect to and pull information from in a very short time frame. So I'm a little confused how to make sure that all 50 links get hit and the script completes successfully before moving on. Thanks in advance!

[kpowning]

In theory, can I do the following and then check header response code for each url and update the url as indexed?

$cnturl = mysql_query("SELECT * FROM DATABASE WHERE `indexed`='0'");

if (mysql_num_rows($cnturl)){

do {

    $geturls = mysql_query("SELECT * FROM DATABASE WHERE `indexed`='0'");

    // CURL RESULT PAGES
    // this is the function called. If the header response for the url == 200 then url gets updated in the database as `indexed`='1' if not then url is not update, therefore do / while
    $rc = new RollingCurl("get_urlinfo");
    $rc->window_size = 10;
    foreach ($geturls as $trpage) {
    $request = new RollingCurlRequest($trpage);
    $rc->add($request);
    }
    $rc->execute();	

} while(mysql_num_rows($geturls));

}

PLEASE - any feedback greatly appreciated. I think this will work nicely, but just want to ask the experts 🙂 I have 870 urls on one site I have to query as fast as possible and make sure all pages are actually gathered