Using CURL within a loop to download a file, 1st one works, 2nd one times out

[kitenski]

Morning all,

I am using CURL to download an image file within a loop. The first time it runs fine and I see the image appear in the directory.

The second time it fails with a timeout, despite it being a valid URL.

Can anyone suggest why it always fails on the 2nd time and how to fix it?

The snippet of code is:

// download image 

$extension = "gif"; 
$ch = curl_init(); 
curl_setopt($ch, CURLOPT_TIMEOUT, 90); 
curl_setopt($ch, CURLOPT_URL, $imgurl); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); 
echo $imgurl . " attempting to open URL "; 
$i = curl_exec($ch); 
if ( $i==false ) { 
  echo curl_errno($ch).' '.curl_error($ch); 
} 
$image_name=time().'.'.$extension; 
$f = fopen('/fulldirectorypath/' . $image_name ,'w+'); 
fwrite($f,$i); 
fclose($f);

I have put an echo in there to display the $IMGURL, to check it is valid, and upped the timeout to 90 secs, but it still fails. I have also added a 240 sec sleep within the loop, but I still get the same error.

This is what I see on screen:

http://images.eu-xmedia.de/thumbnails/34555861/5676051/pt=pic,lang=2,origfile=yes/image.gif attempting to open URL 28 Operation timed out after 90 seconds with 0 bytes received

an empty file is created in my directory.

thanks alot,

Greg

[bradgrafelman]

Is all of that code placed in the loop? If so, try moving the initialization part before the loop, so that the only thing that you do inside the loop is call curl_exec() and process the download.

[kitenski]

hi there,

Thanks for the quick response, yes it is all inside the loop.

$imgurl will change for each iteration of the loop, does it then matter if

curl_setopt($ch, CURLOPT_URL, $imgurl);

is outside of the loop?

[bradgrafelman]

Ah, yes, you wouldn't want that outside of the loop if the URL is going to change. You would want to leave that line inside the loop to update the cURL resource with the appropriate URL, of course.

If that doesn't work, do you have error_reporting set to E_ALL?

[kitenski]

yes, the first lines of my php file are:

<?php 
 error_reporting(E_ALL); 
 ini_set("display_errors", 1); 
?>

and it still doesn't work, even if I move some bits out so the code now looks like ( I also added the curl_close to see if that helped )

// download image

curl_setopt($ch, CURLOPT_URL, $imgurl);
echo $imgurl . " attempting to open URL ";
$i = curl_exec($ch);
if ( $i==false ) {
  echo curl_errno($ch).' '.curl_error($ch);
}

$image_name=time().'.'.$extension;
$f = fopen('/path/to/o/' . $image_name ,'w+');
fwrite($f,$i);
fclose($f);
curl_close ($ch);

[bradgrafelman]

Did you call curl_close() inside the loop? If so, you'll have to call curl_init() again (and set all of the options), as curl_close() clears everything.

Can you show us more of the code you're using? Also, I'm assuming you're still getting the same cURL error number/message, correct?

[kitenski]

Interestingly just worked out if I create a single write using a different IMG URL it works.

If I then do a single write with the problematic one, it fails...

So this fails

<?php 
 error_reporting(E_ALL); 
 ini_set("display_errors", 1); 

$imgurl='http://images.eu-xmedia.de/thumbnails/34555861/5676051/pt=pic,lang=2,origfile=yes/image.gif';
$extension = "gif";

$ch = curl_init();
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_URL, $imgurl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$i = curl_exec($ch);
if ( $i==false ) {
  echo curl_errno($ch).' '.curl_error($ch);
}
$image_name=time().'.'.$extension;
$f = fopen('/path/to/m/' . $image_name ,'w+');
fwrite($f,$i);
fclose($f)
?>

whereas this works

<?php 
 error_reporting(E_ALL); 
 ini_set("display_errors", 1); 

$imgurl='http://images.eu-xmedia.de/thumbnails/246380/5710261/pt=pic,lang=2,origfile=yes/image.gif';
$extension = "gif";

$ch = curl_init();
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_URL, $imgurl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$i = curl_exec($ch);
if ( $i==false ) {
  echo curl_errno($ch).' '.curl_error($ch);
}
$image_name=time().'.'.$extension;
$f = fopen('/path/to/m/' . $image_name ,'w+');
fwrite($f,$i);
fclose($f)
?>

[kitenski]

just realised I am posting the full directory path, is that a security risk? I can't find a way to edit my posts.....

[bradgrafelman]

That's odd.. both URLs work fine in a browser, but the first code snippet fails on your server, while the second one works just fine? Try using [man]file_get_contents/man instead of using cURL (if that's an option?).

kitenski wrote:

just realised I am posting the full directory path, is that a security risk?

I wouldn't think so... if someone's compromised your site/system, not knowing the directory path is probably only going to slow them down by a few seconds. :p Regardless, I obfuscated the paths in your posts above.

[zypher11]

error_reporting(E_ALL);
ini_set("display_errors", 1);

make sure your changing those settings in your php.ini file. The error reporting methods are passed when the file is compiled so these functions don't actually execute till after that. A fatal error in the page would completely ignore these error setting escalations. Kind of defeats the purpose of even having them.

[bradgrafelman]

Since there are no parse errors in the script, using [man]error_reporting/man at the top of the script itself is fine...

[zypher11]

Instead of running each one individually maybe attempt to use the curl_multi_add_handle and curl_multi_exec functions. Use the loop to simply create all the options and variables then run it all at once? Definitely not the most experienced person at cURL but figured id toss out my 2cents

http://www.rustyrazorblade.com/2008/02/curl_multi_exec/
-tutorial