How do I download a file and unzip it with curl ?

[habbardone]

Hello,

What I want to do is download file like this:
www.support-focus.com/test_map.zip

This is my sitemap zipped up and I want to use this as a test.

I am going to run the php script on a different server.

test_map is an xml file so what I want to do is download
the test_map.zip, unzip it and rename it to sitemap.xml

When I googled this, I found reference to a Pclzip class
But I was thinking there might be something in the curl
library that does it.

Since I want to grab the file with curl, does anyone know if curl can
unzip the file ?

If so how do I do it with the above file.

many thanks

[BuzzPHP]

It looks like Curl supports gzip "compression", which is different to a zip file.

gzip is just a encoding. I dont think its a multi file format in this context.

I do recall making tar balls and the gzip-ing them, but thats different to "zip" files.

You can try this API to read your zip file:

http://www.php.net/manual/en/intro.zip.php

Or, you can just store your xml file as a raw xml file and try the gzip encoded transfer.

My understanding is, if you have a transfer between two servers that support the gzip compression, then the source system will compress and transfer the file "compressed", and the destination server will decompress. ie: Fast

Otherwise, it will transfer the data uncompressed. ie: Slow

[habbardone]

Thanks for your reply.
The one I an testing is from my server but the actual one I want to use
is from a third party server that just provides the file in a .zip format.

Does that mean that I may have to do this manually ?

I will have a look at the link you gave.

Thanks

[habbardone]

OK, after a lot of updating of PEAR and stuff I now
have the ZipArchive working ... a bit :o

This is my code:

<?php 

if (!extension_loaded('zip')) {
    dl('zip.so');
}

$func_path =  "/home/guru54gt5/public_html/im/my_functions.php";
require_once("$func_path");

$target_url = "http://www.support-focus.com/test_map.zip";
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';

echo "<br>Starting<br>Target_url: $target_url<br><br>";

// make the cURL request to $target_url
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$page = curl_exec($ch);
if (!$page) {
	echo "<br />cURL error number:" .curl_errno($ch);
	echo "<br />cURL error:" . curl_error($ch);
	exit;
}
curl_close($ch);

// Un zip the file 

$zip = new ZipArchive;
if (! $zip) {
    echo "Could not make ZipArchive object.";
    exit;
}
$zip->open("$page");
$zip->extractTo('./');
$zip->close();
    echo "Ok!";
?> 

The errors I get are in fact warnings:

Warning: ZipArchive::extractTo() [function.ZipArchive-extractTo]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no1.php on line 40

Warning: ZipArchive::close() [function.ZipArchive-close]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no1.php on line 41
Ok!

Line 40 and 41 are:

$zip->extractTo('./');
$zip->close();

Is the problem because my $page is not a file ? 😕

I am a bit confused about this so would appreciate some help.

Thanks.

[dagon]

$page is true or false, its not a file: curl-exec()
var_dump($page);

[BuzzPHP]

ZipArchive :: open() requires a filename. $page is not the destination filename.

Im unsure where your target (source) file will end up on your destination server. I have a feeling $page might actually be the entire zip file stored in a variable, which is a bad thing when it comes to very large zip files.

I found a great tutorial for you:

http://www.tuxradar.com/practicalphp/15/10/2

The following code transfers a file from a URL to a local file:

    $curl = curl_init();
    $fp = fopen("somefile.txt", "w");
    curl_setopt ($curl, CURLOPT_URL, "http://www.php.net");
    curl_setopt($curl, CURLOPT_FILE, $fp);

curl_exec ($curl);
curl_close ($curl);

You then run your ZipArchive code on the destination file. ie: "somefile.txt"

Hope that helps. I would look into writing two functions:

function MyCopyFile($dstfilename, $srcfilename) {}

and

function MyUnzipToFolder($dstpath, $srcfilename) {}

Then, its clearer what your code is doing:

$src = "http://www.mysite.com/file.zip";
$dst = "file.zip";
$path = "temp/";

MyCopyFile($dst, $src);
MyUnzipToFolder($path, $dst);

[habbardone]

Thanks very much for your reply.
I have a look at the tute and will spend some more time
on it later today 🙂

I have progressed the script but I still
get errors and it does not do the unzipping.

This is what I have:

( I have not put it into functions yet)

<?php 

if (!extension_loaded('zip')) {
    dl('zip.so');
}

$func_path =  "/home/guru54gt5/public_html/im/my_functions.php";
require_once("$func_path");

$target_url = "http://www.support-focus.com/test_map.zip";
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
$file_zip = "new.zip";
$file_txt = "new.txt";

echo "<br>Starting<br>Target_url: $target_url<br><br>";


// make the cURL request to $target_url
$ch = curl_init();
$fp = fopen("$file_zip", "w"); 
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_FILE, $fp);

$page = curl_exec($ch);


if (!$page) {
	echo "<br />cURL error number:" .curl_errno($ch);
	echo "<br />cURL error:" . curl_error($ch);
	exit;
}
curl_close($ch);

echo "<br>Downloaded file: $target_url";
echo "<br>Saved as file: $file_zip";
echo "<br>About to unzip ...";

// Un zip the file 

$zip = new ZipArchive;
    if (! $zip) {
        echo "Could not make ZipArchive object.";
        exit;
    }
    $zip->open("$file_zip");
    $zip->extractTo("$file_txt");
    $zip->close();

echo "<br>Unzipped file to: $file_txt<br><br>";		

?> 

This is my output:

Starting
Target_url: http://www.support-focus.com/test_map.zip

Downloaded file: http://www.support-focus.com/test_map.zip
Saved as file: new.zip
About to unzip ...
Warning: ZipArchive::extractTo() [function.ZipArchive-extractTo]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 52

Warning: ZipArchive::close() [function.ZipArchive-close]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 53

Unzipped file to: new.txt

Although it says that it unzipped the file, it doesn't

Not sure why it is not working 😕

Can anyone see what I have done wrong ?

[BuzzPHP]

mmm, bugger. Ok. So is the command:

$zip->open("$file_zip");

Working?

Check its error code. It should be true on success. ie: === true

The error message "Invalid or unitialized Zip object", seems to suggest that the $zip object is fine, but internally, the ZIP object is incorrect.

I know this because PHP would not say such an error even if it knew $zip should be a ZipArchive class and the open() method doesn't cause an error.

So, Im guessing the open() method isn't working. Check the ZIP is downloaded and valid, and check the error code returned from the open() method.

[habbardone]

Thanks for advice 🙂

I have incorporated 2 changes:

1)The headers have been set to 0 so as not
to put them in the file

2) I have checked to see if open() is working
( It isn't )

Here is my script:

if (!extension_loaded('zip')) {
    dl('zip.so');
}

$func_path =  "/home/guru54gt5/public_html/im/my_functions.php";
require_once("$func_path");

$target_url = "http://www.support-focus.com/test_map.zip";
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
$file_zip = "new.zip";
$file_txt = "new.txt";

echo "<br>Starting<br>Target_url: $target_url";
echo "<br>Headers stripped out";

// make the cURL request to $target_url
$ch = curl_init();
$fp = fopen("$file_zip", "w"); 
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_HEADER,0); 
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_FILE, $fp);

$page = curl_exec($ch);


if (!$page) {
	echo "<br />cURL error number:" .curl_errno($ch);
	echo "<br />cURL error:" . curl_error($ch);
	exit;
}
curl_close($ch);

echo "<br>Downloaded file: $target_url";
echo "<br>Saved as file: $file_zip";
echo "<br>About to unzip ...";

// Un zip the file 

$zip = new ZipArchive;
    if (! $zip) {
        echo "<br>Could not make ZipArchive object.";
        exit;
    }
    if($zip->open("$file_zip") != "true") {
		    echo "<br>Could not open $file_zip";
				}
    $zip->extractTo("$file_txt");
    $zip->close();

echo "<br>Unzipped file to: $file_txt<br><br>";		

?> 

And the result:

Starting
Target_url: http://www.support-focus.com/test_map.zip
Headers stripped out
Downloaded file: http://www.support-focus.com/test_map.zip
Saved as file: new.zip
About to unzip ...
Could not open new.zip
Warning: ZipArchive::extractTo() [function.ZipArchive-extractTo]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 55

Warning: ZipArchive::close() [function.ZipArchive-close]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 56

Unzipped file to: new.txt

So the ope() is not working - but I do not know why 😕

The file is only a small sitemap file zipped up.

So I opened the downloaded file new.zip with notepad

This is what it contains:

PK Z‚O;hm×M› 2 test_map.xmlÅWÛrÚ0}ç+^;‹MnMâaš™¶Éz{ʨò‚5µµ®$cøû®Ié´Mi+¿X2»:Çg/^{†W번Vh¬"}Ùí÷’n„ZR¦ôò²ûn>óîUÚÖ¦°è:Q{±‹¶—ÝܹjÇMÓô¬rXŠÊöÈ,c+sÞØx÷0NzÝ}ÏÁÚª¼›ãÖï(IúñÇׯf?(mÐ}l~ER¸–ï(ìNÙOð{X÷Ö6ë²Ï} "iP8Ì¢F¹<šÄèVJc4ÛÚFSÔh„#y~kx„‘TF i§Õ4eVÂdºO¨®2$ëÖ<ÆÞÄ›Ê\è%.~M3¡ŠÍ0Þ{â
£È(·I9’É0~Üv†q÷ûJg¸îUyõ'èIïü0ô†Ì]%@0|[¡T¢†/I;!]0ü‚–J‡SWð¹+‹ï
a]IYÊ]êú ôOçýãÁÉÙàôèy’|µ=˜üg¾M´VVBn‚á£0škÕÞgÊÊB¨M0.Â8%ŒÄÈÝWÂÀáìä ãÚ:â(ÀÍJI„-#cô!ßÀŒ4Üh®ÍëÊÚÇ‹CíÌæ9ÂDp³µœ8ð]m4§¹Bî%¨ÙØP sâ¶TÃÅuî¹]j,O‹^{H‚ëz³]ûÈ"|¢šÃ.¸`Lz7U#<Ã¥ãÞ æktOG˜3}"Þý’t8™§j…0õ&üTI#myÿE8w†Øéæ€ö¿ªyµŸÕ-LÈÀvLæ¼±|أóû»]<äñè³·âdQîjÃè–“â%5>YÆ-—"û662»}{¯xL!&ý†œçïÓ¼ýs!«¥ûëÚúÿL¿PK Z‚O;hm×M› 2 test_map.xmlPK : Å

Any ideas why things are not working ?

[Installer]

ZipArchive->open() takes a path, not file, argument. The extracted files will be placed in that path.

"new ZipArchive" does not return true or false, since if ZipArchive doesn't exist the script will exit with a fatal error. Check for the existence of the class first.

ZipArchive->open() returns boolean true on success, not the string "true". Also if it fails it will return an integer error code which will resolve to true unless a strict comparison is made (!== or ===). You are making an unstrict comparison (!=).

The function ZipArchive->getStatusString() will return a string describing the results of the latest operation (a file must have been successfully opened first).

Both ZipArchive->extractTo() and ZipArchive->close() return either true or false. The former does not necessarily return false if the extraction fails. If the latter returns false, you know that the operation failed.

Try something like this (works for me):

// After curl code:
fclose($file_zip);

$file_dir = './zip-test/';

if (!class_exists('ZipArchive')) {
    echo '<br />Class "ZipArchive" not found';
    exit;
} else {
    $zip = new ZipArchive;
}

if (($err = $zip->open($file_zip)) !== true) {
    echo '<br />Could not open ' . $file_zip;
    echo '<br />Error #' . $err;
    exit;
} else {
    echo '<br />File opened';
}
echo '<br />StatusString: ' . $zip->getStatusString();

if (!$zip->extractTo($file_dir)) {
    echo '<br />Extraction error';
}
echo '<br />StatusString: ' . $zip->getStatusString();

if (!$zip->close()) {
    echo '<br />Error on close';
} else {
    echo '<br />Unzipped file to: ' . $file_dir;
}

[habbardone]

Thanks for your detailed reply.

I have implemented the extra details.

I now get this result:

Starting
Target_url: http://www.support-focus.com/test_map.zip
Headers stripped out
Downloaded file: http://www.support-focus.com/test_map.zip
Saved as file: new.zip
About to unzip ...
File opened
Fatal error: Call to undefined method ZipArchive::getStatusString() in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 66

It is a lot closer 🙂

This is the code I have:

After the curl download ...

curl_close($ch);

echo "<br>Downloaded file: $target_url";
echo "<br>Saved as file: $file_zip";
echo "<br>About to unzip ...";

// Un zip the file 

// After curl code:
fclose($fp);

$file_dir = './zip-test/';

if (!class_exists('ZipArchive')) {
    echo '<br />Class "ZipArchive" not found';
    exit;
} else {
    $zip = new ZipArchive;
}

if (($err = $zip->open($file_zip)) !== true) {
    echo '<br />Could not open ' . $file_zip;
    echo '<br />Error #' . $err;
    exit;
} else {
    echo '<br />File opened';
}
echo '<br />StatusString: ' . $zip->getStatusString();

if (!$zip->extractTo($file_dir)) {
    echo '<br />Extraction error';
}
echo '<br />StatusString: ' . $zip->getStatusString();

if (!$zip->close()) {
    echo '<br />Error on close';
} else {
    echo '<br />Unzipped file to: ' . $file_dir;
} 

?> 

Seems to not like the getStatusString()

I don't know why, it appears in the manual ?

Any suggestions ?

[Installer]

You probably have PHP version < 5.2.7 (Look at the top of the manual page for getStatusString()). Just remove the lines that call that function, and it should work.

Edit: BTW, that manual page says the function "Returns a file pointer (resource) on success or FALSE on failure." I think that's wrong; I think it returns a string in either case (the declaration given there is "string ZipArchive::getStatusString ( void )"). Otherwise it's badly misnamed. :rolleyes: