[RESOLVED] Cross Server File Transfer

[cesarcesar]

I need to transfer files between a network of servers all over the world. I have looked into using cURL and ftp_fput() but am having little luck. I can get a transfer going from my local to the server or visa versa but not from server to server.

Basically im trying to replicate Akamai on a very small scale. This is the scenario. I am working from home. I search for a file on Web Server 1 in Oregon. The file doesn't exist. I search my other servers and find that it is located on Server 2 in Germany. I then need to COPY the file from Web Server 2 to Web Server 1. That's it.

I have it all in place except getting the server-to-server transfer to work. Who can help? Thanks much.

[cesarcesar]

One other thing. The connections need to be made via FTP. I have IP, USER & PASSWORDS for each account. Thanks.

[sneakyimp]

[man]cURL[/man] supports FTP and is probably the easiest way to connect to anything using FTP in PHP. Read the manual (and the comments) and you'll probably find a good FTP example.

If cURL simply cannot connect from machine to machine, there may be a permission problem or firewall problem or server restrictions that will prevent you from connecting. This would probably mean you are out of lucky using cURL or anything else.

[cesarcesar]

Thanks for all your input. Let me explain a little more about what i have done. First of, each of the files I'm transferring are over 1Gb so file_get_contents() wont work. Files are not transferred until requested by the user. I would prefer to do it all in CURL, but as i said im having issues.

I have looked into SST scripts but none seems to satisfy.

My thinking is that cURL cannot directly transfer a file server to server (please correct if wrong). I thought that the way to make it work would be to call a file on Web Server 2 using cURL and add some parameters (file name to transfer) to the call so that when ran it would using cURL transfer the file to Web Server 1. Sound good? Problem, Web Server 2 doesn't have a URL that will resolve to a specific file on the server. Therefor again only cURL via FTP login can be used (as i see it).

So do further define my question, with cURL, is there a way to transfer files between servers directly using a curl statement like

$ch = curl_init("ftp://theuser:thepass@72.47.251.180/testfile.m4v");

but one that works more like below with a location to get and set from/to. I know the following code is incorrect, i just would like to know how to get it right.

$c = curl_init("ftp://theuser:thepass@72.47.251.180/testfile.m4v","ftp://theuser:thepass@42.76.151.236/testfile.m4v");

I know i have left out parts of the cURL process. curl_exec() and curl_close() are pretty standard. Thanks.

[sneakyimp]

You need to read the curl documentation. curl usually requires that you have several lines to set various options. Check out [man]curl_setopt[/man]. Googling also helps. I found this in about 30 seconds:
http://www.web-development-blog.com/archives/tutorial-ftp-upload-via-curl/

[cesarcesar]

I know about the "other lines" as i stated in my last post. The other lines are pretty standard. If i need an extra curl_setopt() then thats great.. but what is it.

The link you sent is for uploading from ones PC to a server. Not what im looking for. thanks for the effort though.

[sneakyimp]

You obviously aren't reading anything I suggest in any real detail. Had you read that example (the very first one you get from googling 'php curl ftp example') then you would have seen this code at the bottom:

 	$ch = curl_init();
 	$localfile = $_FILES['upload']['tmp_name'];
 	$fp = fopen($localfile, 'r');
 	curl_setopt($ch, CURLOPT_URL, 'ftp://ftp_login:[e-mail]/'.$_FILES['upload']['name']);
 	curl_setopt($ch, CURLOPT_UPLOAD, 1);
 	curl_setopt($ch, CURLOPT_INFILE, $fp);
 	curl_setopt($ch, CURLOPT_INFILESIZE, filesize($localfile));
 	curl_exec ($ch);
 	$error_no = curl_errno($ch);
 	curl_close ($ch);
        if ($error_no == 0) {
        	$error = 'File uploaded succesfully.';
        } else {
        	$error = 'File upload error.';
        }

Hopefully that will help you. If not, try reading some of the other google results.

[bradgrafelman]

I think the general procedure you need to do is first use cURL to download the file from Server 2 onto the computer running this script - a temp directory, for example. Break that connection and then connect to Server 1 and use code similar to what sneakyimp posted above to upload the (now locally-stored) file to Server 1, then remove the local copy of the file.

What this does is effectively perform a server-to-server FTP transfer. Now, I myself have never done this, but it's supposedly possible to do exactly that - a server-to-server FTP transfer. Here is an example of a Microsoft KB article talking about doing so using IIS, but I would think that the process is similar with other FTP servers. Note, however, that server-to-server transfers isn't usually enabled by default, so you might have to contact your host to see if this is even possible.

[cesarcesar]

@ - thank you much for your suggestion. You where correct. I just needed to pay a little more attention to the specific link you sent. I DID look at it, I just thought (wrongly) that it was not exactly what i needed. Thank you again.

I do have to say though that your delivery sucks. I'm not sure if you have been in the forums way too long and have answered "every dumb question" under the sun, but your lack of patience with the "little people" is showing in your comments. Maybe take a step back and assume first that people to Google what there looking for, and that people have tried other scripts before asking advice. Remember you are in the forum to help... not belittle.

[cesarcesar]

SOLVED -

Thanks for everyones help. I am posting my full function for those who may find it useful. To recap, this function "try's" to simulate Akamai's service.

1. GeoLoc user.
2. GeoLoc all servers
3. Calculate distance between user and each server.
4. Sort by distance.
5. Check to see if file is at closest server. If so download.
6. If file not at closest server, copy file from mother server to server missing file. Then notify user via email with download link to closest server.

function sst() {

global $_GET, $_SERVER;

/*   start geo loc   */
// http://www.imaginerc.com/software/GeoCalc/
require_once('standards/classes/geoplugin.class.php'); 
$geoplugin = new geoPlugin();
// http://www.phpclasses.org/browse/file/25516.html
include_once("standards/classes/GeoCalc.class.php"); 
$oGC = new GeoCalc();

/*   set server ip's and FTP logins		*/
$servers = array($_SERVER['REMOTE_ADDR'],'xxx.xx.xx.xxx','xxx.xx.xx.xxx');
$user = array('','usr1','usr2');
$pass = array('','pass1','pass2',);

/*   get user location   */
$geo[0]['ip'] = $servers[0];
$geoplugin->locate($geo[0]['ip']);
$geo[0]['lat'] = $geoplugin->latitude;
$geo[0]['lon'] = $geoplugin->longitude;
$geo[0]['city'] = $geoplugin->city;
$geo[0]['countryName'] = $geoplugin->countryName;

/*   get  servers data  */
for ($i = 1; $i < count($servers); $i++) {

	$geo[$i]['ip'] = $servers[$i];
	$geo[$i]['user'] = $user[$i];
	$geo[$i]['pass'] = $pass[$i];
	$geoplugin->locate($geo[$i]['ip']);
	$geo[$i]['lat'] = $geoplugin->latitude;
	$geo[$i]['lon'] = $geoplugin->longitude;
	$geo[$i]['city'] = $geoplugin->city;
	$geo[$i]['countryName'] = $geoplugin->countryName;
	$geo[$i]['dist'] = $oGC->GCDistance($geo[0]['lat'],$geo[0]['lon'],$geo[$i]['lat'],$geo[$i]['lon']);

}

/*   order by distance.   */
$geo = orderBy($geo, 'dist');

/*   connect to closeest server   */
$conn_id = ftp_connect($geo[1]['ip']) or die("Couldn't connect to $ftp_server");
ftp_login($conn_id, $geo[1]['user'], $geo[1]['pass']);
$contents = ftp_nlist($conn_id, "tts/incoming");
ftp_close($conn_id);

$localfile = '/user/dac15/tts/incoming/';
$remotefile = '/user/'.$geo[1]['user'].'/tts/incoming/';

/*  if file is on the closest server.    */
if (find($_GET['name'],$contents)) {

	// http://www.awesomephp.com/?Tutorials*16/Download-file-with-resume,-stream-and-speed-options.html
	downloadFile($remotefile,$_GET['name'],900,false); 

}else{ 

	/*   push file to the closest server requesting server.*/
	$ch = curl_init();
	$fp = fopen($localfile.$_GET['name'], 'r');
	curl_setopt($ch, CURLOPT_URL, 'ftp://'.$geo[1]['user'].':'.$geo[1]['pass'].'@'.$geo[1]['ip'].'/'.$remotefile.$_GET['name']);
	curl_setopt($ch, CURLOPT_UPLOAD, 1);
	curl_setopt($ch, CURLOPT_INFILE, $fp);
	curl_setopt($ch, CURLOPT_INFILESIZE, filesize($localfile.$_GET['name']));
	curl_exec($ch);
	$error_no = curl_errno($ch);
	curl_close($ch);
	if ($error_no == 0) {
		$error = 'File uploaded succesfully.';

		$mailto[0][0] = $_SERVER['HTTP_EMAIL'];
		$mailto[0][1] = $_SERVER['HTTP_FNAME'];
		$mailto[0][2] = $_SERVER['HTTP_LNAME'];

		$content="
		Dear ".$mailto[0][1]."
		<br>
		Your file is ready for download. Go to this link in your preferred web browser.
		";

		// http://www.xpertmailer.com/
		if (smtp_mail($mailto,"DoD File Download",$content)) { Return 1; }

	}else{
		$error = 'File upload error.';
		echo $error;
	}

}

die;

}

function orderBy($data, $field, $reverse_sort='') {
	$code = "return strnatcmp(\$a['$field'], \$b['$field']);";
	usort($data, create_function('$a,$b', $code));
	if ($reverse_sort==1) { krsort($data); }
	return $data;
}

[bradgrafelman]

Glad to hear you got it sorted! Don't forget to mark this thread resolved (using the link on the Thread Tools menu above).

[cesarcesar]

I have another question to go along with this post. Once the cURL transfer is started on my 1GB file the screen just sits white while doing its thing (page load bar crawls). Is there a way to finish loading the page while the transfer is in process. This way the user doesn't have to wait for the transfer to finish, they are just there to make it happen. (they are not downloading at the time so it would be okay for them to continue). Thanks.

[sneakyimp]

I meant to mention that. 1gb is an enormous file to be transferring. If you want the transfer to keep going even if the user navigates away or closes their browser, you are probably going to need to use [man]ignore_user_abort[/man]. You might want to try outputting your entire page first including all the html you would like to show and then doing the transfer as the very last thing on the page. I'm not certain, but I think a browser would still say "transferring data" in the status bar so it would look like the page was still loading. Another possible bit of weirdness is that the web server might not bother sending the page html until a certain interval of time. I've noticed that when a really long script is running and outputting a very large amount of html that the html arrives in chunks. The timing is unpredictable sometimes. I'm not sure how to deal with that.

Another possiblity which I like better is that you can put all the code that does the file transfer into a separate php file and then execute it using exec. If you put an ampersand (&) after the command to execute the php file then linux will fork off a process to execute the command and return immediately. Your script to transfer the file might look like this:

// call this file transfer.php

$file = $argv[1];
$server = $argv[2];

echo "Transfer of " . $file . " to " . $server . " initiated";

// put code to transfer the file $file to the server $server here

And then you could put this in your page script. It will launch transfer.php in the background.

echo "initiating transfer script\n";
exec('php transfer.php /path/to/file.ext server/goes/here > output.txt &', $output, $result);

if ($result) {
        die('transfer script failed!');
}

[cesarcesar]

@ - very smart suggestion. thank you i will give it a shot and reply with results.

[cesarcesar]

/**
 * Takes a needle and haystack (just like in_array()) and does a wildcard search on it's values.
 *
 * @param    string        $string        Needle to find
 * @param    array        $array        Haystack to look through
 * @result    array                    Returns the elements that the $string was found in
 */
function find ($string, $array = array ())
{
	foreach ($array as $key => $value) {
		unset ($array[$key]);
		if (strpos($value, $string) !== false) {
			$array[$key] = $value;
		}
	}
	return $array;
}