Resource Hog?

rdmustang

I have a script that uses curl to download a webpage and waits for it to finish.

Code snipit:

  curl_setopt($ch, CURLOPT_URL, $url );
  curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
  curl_setopt($ch, CURLOPT_POST, 0 );
  curl_setopt($ch, CURLOPT_HEADER, 0);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);
  $response = curl_exec( $ch );

  $mh = curl_multi_init();
  curl_multi_add_handle($mh,$ch);
  $running=null;
  curl_multi_exec($mh,$running);
  while ($running > 0) {
    curl_multi_exec($mh,$running);
    sleep(5);
  } ;

If users1 requests this page on webserver1 then it starts and usually takes a few minutes (the page it's requesting is on webserver2 and the page on webserver 2 takes a while to finish).
Then while user1 is waiting, if user2 comes along and requests any other page on webserver1 then user2 must wait until user1's page finishes.

I've done some testing and it only seems to happen with the page containing the code above. While user1 is requesting this page, 1.3% of the CPU is used (httpd is using the 1.3%) and only 261mb of 1024mb of RAM is being used (httpd only taking up up 1.6% of the RAM).

I'm a newbie to PHP.. The only thing I've tried is upping the allowed memory to 128M in php.ini (good or bad idea?).

Any suggestions?

Kudose

That just tells PHP to allow each running script to consume 128M before it throws a fatal error.

I'm not 100% sure what you can do here, but you could try a do while loop and close your connections after each call. http://us.php.net/manual/en/function.curl-multi-init.php

Also, why are you creating $response when you don't use it?

rdmustang

Kudose;10890437 wrote:
That just tells PHP to allow each running script to consume 128M before it throws a fatal error.

I'm not 100% sure what you can do here, but you could try a do while loop and close your connections after each call. http://us.php.net/manual/en/function.curl-multi-init.php

Also, why are you creating $response when you don't use it?

I use $result later on. The reason for the connection is to grab the output of a page on a remote web-server. I can't close the connection because then I wouldn't receive the output. Right?

Kudose

Unless the script it constantly outputting new data (the page would never stop loading), a normal curl call is fine. The purpose to the multi function is to run multiple curl calls at once.

If your page doesn't ever stop streaming data, you should use socket functions or file I/O functions instead.

rdmustang

I changed it over to a regular curl and it's still doing the same thing. While 1 user is waiting for the remote page to load no other pages on the site will load until the curl finishes.

Any ideas? 🙂

Kudose

The only reasons that would be the case (I can think of) is if there are not enough resources on the server or if you are accessing database tables with locks on them and the script is waiting for lock releases.