Uploading and resizing an image bigger than 200k, how do sites like Flickr do it?

[SpockBert]

I've got a script that allows a signed in user to browse their computer and upload an image. The script then resizes the uploaded image, once for the correct display size and secondly for a thumbnail.

It all works fine but.....only for smallish images 🙁

By trial and error any image bigger than 200k tends to break the entire script and I get an ugly error page.

I tried fixing it by adding a size filter:
//check size
if ($_FILES["file"]["size"] > 200000){
header("Location: http://www.myDomain.com/tooBigError.php");
exit();
}

That kind of works (sometimes even a 194k file trips the filter) but 200k is ridiculously low, especially when the users I'm targetting are likely to be taking their pictures with hi-res digital cameras. Also anything way bigger than 200 completely breaks the script and I get an error again! Sometimes I get a php memory error too.

I've tried all sorts of things but short of directing the users to an image resize site (an ugly solution) I'm out of ideas. Ideally I'd like to be able to upload images of 3Mb.

I'm on a decent but shared hosting package, just wondering how Flickr and other sites easily upload large images with no problems?

Anybody know? All ideas welcome!

[planetsim]

Can you elaborate on what the error is. From what I can tell your script is doing some crazy stuff it would help if you would show it rather than a simple if redirect if its bigger than 200k. My best guess would be the host with some hard memory limits and file upload limits but Im only guessing.

[james_h]

Hi SpockBert,

There are limits set in the php.ini, these normally differ from host to host and on shared hosting the limits can be somewhat restricting (to ensure 1 site doesn't slow down other sites).

You may be able to change these, if your hosts configuration supports.

There are 3 limits i think you will encounter:
post_max_size (the max size of the POST request - the entire form submission)
upload_max_filesize (the max size of an individual file included in the POST request)
memory_limit (the maxmimum amount of memory your script can consume, large images require large amounts of memory to resize)

I think post_max_size, upload_max_filesize can be set during runtime (i.e. at the start of your code), like this:

ini_set('post_max_size', '8M');
ini_set('upload_max_filesize', '8M');

However i think memory_limit can only be changed BEFORE runtime. You may be able to do this via a .htaccess. Create a file called ".htaccess", containing the following line, and place it in the same directory as your scripts. You can also use htaccess to set post_max_size+upload_max_filesize (again, if it works at all).

php_value memory_limit 24M

If it works, you'll be able to see the results using the phpinfo() function.
Create a test script containing

<?
echo phpinfo();
?>

Hope this helps.
James_h

[SpockBert]

Thanks for the replies guys. Basically I get 2 errors...

A) Any image more than 1500 pixels wide generates a memory error:

Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 6400 bytes) in /vhost/vhost6/c/a/m/mydomain.com/www/assets/myUploadScript.php on line 199

😎 Any image with a filesize greater than 510k gets a "Problem loading page: myUploadScript.php not found/unavailable etc etc"

James, thanks for your suggestion, I used a .htaccess file to change the memory allocation (pumped it up from 8M to 64M, that too high?) and FANTASTIC....I can now upload and resize images as large as 3000px, but...

and here's a killer ONLY if that image is no greater than 510k, I've tried your suggestion of:

ini_set('post_max_size', '8M');
ini_set('upload_max_filesize', '8M');

In the beginning of my "myUploadScript.php" but it didn't change anything.

Thanks for the help guys, I'm halfway there on this problem now, if I can somehow raise the upload limit above 500k I've cracked it.

Is there any more I can do to the .htaccess to lick that problem do you think?

[DaemonProjects]

you have to split the file into chunks. then take the chunks and combine them back.

Checkout - http://www.jumploader.com/
Partitioned upload - http://www.jumploader.com/doc_serverside_partphp.html

[SpockBert]

Thanks Daemon

I'll try that method if all else fails, but I think I definitely know what the problem is now:

From my own testing it seems anything greater than 510k breaks the site well from this site: http://www.radinks.com/upload/config.php check this out:

Apache Settings

The apache webserver has a LimitRequestBody configuration directive that restricts the size of all POST data regardless of the web scripting language in use. Some RPM installations sets limit request body to 512Kb. You will need to change this to a larger value or remove the entry altogether.

Changing that LimitRequestBody value is probably beyond the scope of .htaccess files, I'm guessing you need access to the server's Apache config file and I'm never gonna get near that on a shared server so I don't know how to solve this 🙁

[SpockBert]

Hey Daemon

That Jumploader thing, do you use it yourself?

Only I've having a nightmare getting it to work, is it difficult to setup? I can see my applet, it uploads the image but when it gets to 100% it gets a failed message but thats it. I've tried CMMODing the folder to 777, also how you supposed to tell it where to upload to?

Cheers!

[DaemonProjects]

ya I use it all the time. I'll post my backend code

[DaemonProjects]

upload.php

<applet
		code="jmaster.jumploader.app.JumpLoaderApplet.class" name="jumpLoaderApplet"
		width="700"
		height="400" 
		mayscript="mayscript"
		archive="jumploader_z.jar" id="jumpLoaderApplet">
      <param name="uc_uploadUrl" value="transfer.php?category=<? echo $_POST["c"]; ?>"/>

transfer.php

<?
//----------------------------------------------
//    partitioned upload file handler script
//----------------------------------------------

//
//    specify upload directory - storage 
//    for reconstructed uploaded files
$upload_dir = "uploaded/".$_REQUEST["category"]."/";

//
//    specify stage directory - temporary storage 
//    for uploaded partitions
$stage_dir = "uploaded/stage/";

//
//    retrieve request parameters
$file_param_name = 'file';
$file_name = $_FILES[ $file_param_name ][ 'name' ];
$file_id = $_POST[ 'fileId' ];
$partition_index = $_POST[ 'partitionIndex' ];
$partition_count = $_POST[ 'partitionCount' ];
$file_length = $_POST[ 'fileLength' ];

//
//    the $client_id is an essential variable, 
//    this is used to generate uploaded partitions file prefix, 
//    because we can not rely on 'fileId' uniqueness in a 
//    concurrent environment - 2 different clients (applets) 
//    may submit duplicate fileId. thus, this is responsibility 
//    of a server to distribute unique clientId values
//    (or other variable, for example this could be session id) 
//    for instantiated applets.
$client_id = $_GET[ 'clientId' ];

//
//    move uploaded partition to the staging folder 
//    using following name pattern:
//    ${clientId}.${fileId}.${partitionIndex}
$source_file_path = $_FILES[ $file_param_name ][ 'tmp_name' ];
$target_file_path = $stage_dir . $client_id . "." . $file_id . 
    "." . $partition_index;
if( !move_uploaded_file( $source_file_path, $target_file_path ) ) {
    echo "Error:Can't move uploaded file";
    return;
}

//
//    check if we have collected all partitions properly
$all_in_place = true;
$partitions_length = 0;
for( $i = 0; $all_in_place && $i < $partition_count; $i++ ) {
    $partition_file = $stage_dir . $client_id . "." . $file_id . "." . $i;
    if( file_exists( $partition_file ) ) {
        $partitions_length += filesize( $partition_file );
    } else {
        $all_in_place = false;
    }
}

//
//    issue error if last partition uploaded, but partitions validation failed
if( $partition_index == $partition_count - 1 &&
        ( !$all_in_place || $partitions_length != intval( $file_length ) ) ) {
    echo "Error:Upload validation error";
    return;
}

//
//    reconstruct original file if all ok
if( $all_in_place ) {
    $file = $upload_dir . $client_id . "." . $file_id;
    $file_handle = fopen( $file, 'a' );
    for( $i = 0; $all_in_place && $i < $partition_count; $i++ ) {
        //
        //    read partition file
        $partition_file = $stage_dir . $client_id . "." . $file_id . "." . $i;
        $partition_file_handle = fopen( $partition_file, "rb" );
        $contents = fread( $partition_file_handle, filesize( $partition_file ) );
        fclose( $partition_file_handle );
        //
        //    write to reconstruct file
        fwrite( $file_handle, $contents );
        //
        //    remove partition file
        unlink( $partition_file );
    }
    fclose( $file_handle );

// Rename File
rename($file,$upload_dir.strtolower($file_name));
}
?>

make sure the directory upload/ and upload/stage/ is chmod 777 (writable)

[SpockBert]

Thanks Daemon

Wow, I completely misunderstood this applet. I didn't realise you had to write your own script to upload the file, I thought the applet did that itself! I thought that was the whole point of it..haha

I tried using your code, pasted it word for word, works great......but...

sadly just like my original solution it only works for images < 513k, it cannot upload anything bigger than that on my shared server 🙁

Looks like that LimitRequestBody var set on my shared Apache server affects java applets too.

Think I'm stuck now, the (not very helpful) webhosting tech support refused to raise this LimitRequestBody var and suggested upgrading the account to a virtual hosting package.

Thanks for all your help guys, much appreciated.

[bradgrafelman]

1. If you're getting "Allowed memory exhausted..." errors, then your problem has nothing to do with uploading the file - that part has already succeeded and PHP has tried to actually open the file that was POST'ed.

2. You need to increase the memory limit so that PHP can open these larger images - use [man]ini_set/man for this.

3. You can't change post_max_size or upload_max_filesize using [man]ini_set/man within a PHP script; by the time your script is executed, PHP has already decided that the file it has received is too large and has thus discarded it. If these directives need to be adjusted, you'll have to use a .htaccess file.

Also, though I don't know how Flickr works, I do know that an alternative to PHP image resizing is to use 3rd-party software designed (and optimized) at performing this task. One such example is ImageMagick, available on many hosts. All PHP does is execute a command to launch the software - the GD library isn't even needed.