Hi there.

Here's my code, I'm sure I'm doing something wrong in the fread loop but I don't know what.

I have two problems: 1) This takes a really long time top copy large files (taking several minutes to copy a few hundred M😎 and 2) It takes a long time to copy small files too, but most of that is spent re-reading the lsat chunk of the file (as far as I can tell)

What I'm trying to do is make a script that will (eventually, when I can get the thing to work!) be used to backup files from one location to two others at the same time. So that if one drive fails, then I still have a backup on the other destination. I made it clever enough that you just (right now) direct the browser to file.php?src=SOURCE&dest=DEST1&dest=DEST2 and so on, theoretically ad infinitum.

Do you think that I'll ever get this to run on larger files (I've had problems with larger files, >4GB, where only a certain amount of data gets copied and I'm left with a corrupted file in my backup.) and be faster than just execing cp SOURCE DEST1;cp SOURCE DEST2?

I apologise if that made less sense than you would like 🙂

<?php

// set_time_limit (0);	// Stop the script from timing out.
 set_time_limit (1);	// Make the script time out sooner.

 $startTime = time();
 $filesize = 0;

 $arg = explode ('&', substr ($_SERVER['REQUEST_URI'], strpos ($_SERVER['REQUEST_URI'], '?')));

 $_GET['src'] = stripslashes ($_GET['src']);

 $dst = array();

 $chunkNum = 1;

 foreach ($arg as $thisArg)
  {
   if (substr ($thisArg, 0, strpos ($thisArg, '=')) == 'dest')
    {
     $dst[count($dst)] = rawurldecode (substr ($thisArg, (strpos ($thisArg, '=') + 1)));
    }
  }

 if ((isset ($_GET['src'])) && (count ($dst) > 0) && (file_exists ($_GET['src'])))
  {
   if ($sourceFile = fopen ($_GET['src'], 'rb'))
    {
     // Source file is accessible.
     echo 'Reading source!'."\n\n";
     flush();
     foreach ($dst as $thisDest)
      {
       $destFile[count ($destFile)] = fopen ($thisDest, 'wb');
      }
     // If a destFile was not open-able, then it should be silently ignored and the script skips to the next one...
     while (($buffer = fread($sourceFile, 4096)) !== false)
      {
       echo 'Processing 4kb chunk '.$chunkNum."\n";
       flush();
       $chunkNum++;
       foreach ($destFile as $thisDest)
        {
         fwrite ($thisDest, $buffer);
         $filesize += strlen ($buffer);
        }
       /* It seems to freeze here... The files are copied to the new location, and can be opened and viewed, but it
		never releases the lock on the file, and never echos 'Done Copying!' In addition, it seems to be giving
		me many more chunks than it should be... A 2 MB image file copies fine, but with the timeout set to one
		second, it copies the file then reads another 7 MB worth of chunks before being timed out then stopped.		*/
      }
     echo "\n\n".'Done copying!';
     flush();
     fclose ($sourceFile);
     foreach ($destFile as $thisDest)
      {
       fclose ($thisDest);
      }
     echo 'The file '.$_GET['src'].' was successfully copied to '.count ($destFile).' locations.<br />';
     echo $filesize.' bytes (about '.round ($filesize / 1048576).'MB) were copied in '.(time() - $startTime).'seconds. (About '.round((time() - $startTime) / 60).' minutes.)';
     flush();
    }
   else
    {
     echo 'Error: Source file specified ('.$_GET['src'].') could not be accessed by the web server process!';
    }
  }
 elseif (!isset ($_GET['src']))
  {
   echo 'Error: No source file specified! Specify one with &amp;src=<file> in your address bar.';
  }
 elseif (count ($dst) == 0)
  {
   echo 'Error: No destination file(s) specified!';
  }
 elseif (!file_exists ($_GET['src']))
  {
   echo 'Error: Source file specified ('.$_GET['src'].') does not exist!';
  }

?>

    Why not just use PHP's [man]copy/man function?

      NogDog;11019541 wrote:

      Why not just use PHP's [man]copy/man function?

      Sorry for the delayed reply...

      I tried using PHP's copy() function, as you suggested, but for some reason, on that particular machine (Win2K Server SP4, Apache 2.2, PHP 5.2) it didn't work too well... What I mean is that most files copied successfully, but large files (and some of the files generated for this script to move around are DVD-DL ISOs, so can get up to 8 or 9GB!) didn't copy successfully. Off the top of my head, I think that the files in the destination folder ended up being about 200MB in size, so for the last couple of years, I've fallen back to:

      exec ('copy [source] [destination1]');
      exec ('copy [source] [destination2]');

      But as you can imagine when you're dealing with such huge files, then it takes a long time. I'm trying to speed it up by making a system to grab a file and copy it to two locations simultaneously, which will be faster since the machine doesn't have enough RAM to hold the entire ISO in memory for the second copy operation (even if it was likely to try, which I don't think it is!)

      Simple answer so you don't have to think: I tried, it didn't work, but the workaround is very slow and I want it to be faster. 🙂

      I'd write an app in C++ or something similar to do the same thing, then exec() if only I had the expertise to do so! Sadly, I can't seem to find a pre-built app to do this for me 🙁

        What's max RAM set to in php.ini?

        Do you really need to copy instead of simply just move?

          johanafm;11019709 wrote:

          What's max RAM set to in php.ini?

          Do you really need to copy instead of simply just move?

          I can't check max ram right now. The power supply chose a bad time to fail and I've not got the new one yet. Hopefully I'll have that tomorrow or the next day.

          What I'm doing is basically copy, copy, delete, so moving would be an acceptable solution so long as it involved moving to two locations simultaneously.

            Not sure what the performance would be as compared to exec'ing two system commands, but another solution you might try for benchmarking purposes would be:

            1. [man]fopen/man a read stream to the source file.

            2. [man]fopen/man two write streams to the two destinations.

            3. While [man]feof/man is not true on the read stream: [man]fread/man 8192 bytes from the source file and [man]fwrite/man that data twice (once into each of the write streams opened above)

            4. [man]fclose/man all the streams and [man]unlink/man the original source file.

              If it's truly a time-out issue (as oppose to, say, a RAM issue), I might consider using shell_exec() to launch a separate script (maybe even a shell script?) to do the move/copy, and launch it as a background process (nohup....&) so that your PHP script does not have to wait for it. If you want to wait for it, then you might have to use [man]set_time_limit/man to crank up how long your script will wait.

              In either case, you can save some time by doing a rename (move) first, then do just one copy of the renamed file to the 2nd backup file path-name.

                KITTfan2K;11019729 wrote:

                moving would be an acceptable solution so long as it involved moving to two locations simultaneously.

                You can't. If you need data in two places you need at least one copy. Move doesn't move data in the way you probably believe it does - that's the way copy works. Move just says "data is in new place" and then it is - but only as long as it remains on the same disk andas long as you there is no duplication (copy) of data involved. As soon as you "move" from one disk to another, it's really a copy followed by delete.

                Since you need two destination files, I recommend move followed by copy, since this results in one read, one write. Brad's suggestion of (one) fopen source and (two) fopen destinations uses one read, two writes. A plain (naive) copy would mean two reads, two writes.

                  Just another thought: have your script do a move to the first location, and set up a cron job to periodically look for files in that location that are not in the backup location, and copy them over.

                    Write a Reply...