PHP large file uploads

Here I bumped into a really popular subject. My ownCloud had a really small upload limit of 32 MiB and I was aiming for the 1+ GiB range. The "cloud" is in a tiny box and is running a 32-bit Linux, so 2 GiB is the absolute maximum for a file that can pass trough Apache and PHP. The limits are documented in ownCloud Administrators Manual - Dealing with Big File Uploads.

Raising the file size limits is something I could do myself. Here is a reference for you: How to Upload Large Files in PHP. Its simply about finding the parameters for limits and setting them to a bigger value. 

I created different size sample files and tested with them. I found out that there is a point after Apache started the upload, uploaded for a while and exited with a HTTP/500. In my case 600 MiB file passed ok, but 800 MiB file did not. I later found out, that it wasn't about the file sizes itself, but max input time. I had missed that one on my setup.

The max input time is a classic, for example a conversion with topic "PHP file upload affected or not by max_input_time?" discusses the issue in detail. The conclusion is that, the actual upload speed (or network bandwidth available) has nothing to do with the input processing, or maximum value of it. There is a PHP manual page of http://php.net/manual/en/features.file-upload.common-pitfalls.php and it clearly says:

max_input_time sets the maximum time, in seconds, the script is allowed to receive input;
this includes file uploads. For large or multiple files, or users on slower connections,
the default of 60 seconds may be exceeded.

But that simply is not true! In the another section of PHP manual the integer directive max_input_time is defined as:

This sets the maximum time in seconds a script is allowed to parse input data, like POST and GET. Timing begins at the moment PHP is invoked at the server and ends when execution begins.

When is PHP invoked? Let's say you're running Apache. You're actually uploading the file to Apache, which after receiving the file passes the control to a handler. PHP in this case. Surely the input processing does not start at the point where uploading starts.

Test setup

The upload is affected by following PHP configuration directives:

• file_uploads: The master switch. This one is rarely disabled as it makes any file upload processing impossible on PHP.
  
• Changeable: PHP_INI_SYSTEM
• upload_max_filesize: Max size of a single file.
  
• PHP_INI_PERDIR
• post_max_size: Max size of the entire upload batch. A HTTP POST can contain any number of files. In my test only one file is used.
  
• PHP_INI_PERDIR
• max_input_time: As discussed above, the time to parse the uploaded data and files. This would include populating $_FILES superglobal.
  
• PHP_INI_PERDIR
• max_execution_time: The time a script is allowed to run after its input has been parsed. This would include any processing of the file itself.
  
• PHP_INI_ALL
• memory_limit: The amount of memory a script is allowed to use during its execution. Has absolutely nothing to do with the size of the file uploaded, unless the script loads and processes the file.
  
• PHP_INI_ALL 
• upload_tmp_dir: This is something I threw in based on testing. None of the articles ever mention this one. This defines the exact location where the uploaded file initially goes. If the PHP-script does not move the uploaded file out of this temporary location, the file will be deleted when script stops executing. Make sure you have enough space at this directory for large files!
  
• PHP_INI_SYSTEM

A PHP script cannot change all of the introduced configuration values. The changeable limits are defined as:

• PHP_INI_USER: Entry can be set in user scripts (like with ini_set())
• PHP_INI_PERDIR: Entry can be set in php.ini, .htaccess, httpd.conf
• PHP_INI_SYSTEM: Entry can be set in php.ini or httpd.conf
  

For testing purposes I chose the POST and upload max sizes to be 1 GiB (or 1024 MiB). To test the timeout values, I chose relatively small values of 2 seconds both for input parsing and script execution. Also to prove that memory limit does not limit the file upload, I chose the available memory for the script to be 1 MiB. The memory limit is not an issue, as my script does not touch the file, does not try to load or process it.

My test script carefully enforces the above limits just to make sure, that there is no configuration mistakes.

Sample files were generated out of randomness with a command:

dd if=/dev/urandom of=900M bs=1024 count=921600

A number of files of different size was used, but since the POST-limit was set to 1 GiB or 1073741824 bytes, it is impossible to upload a file of the same size. There is always some overhead in a HTTP POST-request. So, the maximum file size I succesfully used with these parameters was 900 MiB. Interestingy it was the 2 second input processing time which caused problems.

The sample code:

<?php
// Adapted by JaTu 2014 from code published in
// http://stackoverflow.com/questions/11387113/php-file-upload-affected-or-not-by-max-input-time

$iniValues = array(
    'file_uploads' => '1',              // PHP_INI_SYSTEM
    'upload_max_filesize' => '1024M',   // PHP_INI_PERDIR
    'post_max_size' => '1024M',         // PHP_INI_PERDIR
    'max_input_time' => '2',            // PHP_INI_PERDIR
    'max_execution_time' => '2',    // PHP_INI_ALL
    'memory_limit' => '1M',         // PHP_INI_ALL
);
$iniValuesToSet = array('max_execution_time', 'memory_limit');
$upload_max_filesize_inBytes = 1073741824;  // 1 GiB

foreach ($iniValues as $variable => $value) {
    $cur = ini_get($variable);
    if ($cur !== $value) {
        if (in_array($variable, $iniValuesToSet)) {
            $prev = ini_set($variable, $value);
            if ($prev === false) {
                // Assume the previous value was not FALSE, but the set failed.
                // None of those variables can reasonable have a boolean value of FALSE anyway.
                die('Failed to ini_set() a value into variable ' . $variable);
            }
        } else {
            die('Failed to ini_set() a value into variable ' . $variable . ' and make it stick.');
        }
    }
}

if (!empty($_FILES) && isset($_FILES['userfile'])) {
    switch ($_FILES['userfile']["error"]) {
    case UPLOAD_ERR_OK:
        $status = 'There is no error, the file uploaded with success.';
        break;
    case UPLOAD_ERR_INI_SIZE:
        $status = 'The uploaded file exceeds the upload_max_filesize directive in php.ini.';
        break;
    case UPLOAD_ERR_FORM_SIZE:
        $status = 'The uploaded file exceeds the MAX_FILE_SIZE directive that was specified in the HTML form.' .
            ' Value is set to: ' . $_POST['MAX_FILE_SIZE'];
        break;
    case UPLOAD_ERR_PARTIAL:
        $status = 'The uploaded file was only partially uploaded.';
        break;
    case UPLOAD_ERR_NO_FILE:
        $status = 'No file was uploaded.';
        break;
    case UPLOAD_ERR_NO_TMP_DIR:
        $status = 'Missing a temporary folder.';
        break;
    case UPLOAD_ERR_CANT_WRITE:
        $status = 'Failed to write file to disk.';
        break;
    case UPLOAD_ERR_EXTENSION:
        $status = 'A PHP extension stopped the file upload. PHP does not provide a way to ascertain which extension caused the file upload to stop; examining the list of loaded extensions with phpinfo() may help.';
        break;
    default:
        $status = 'No idea. Huh?';
    }

    print "Status: {$status}<br/>\n";
    print '<pre>';
    var_dump($_FILES);
    print '</pre>';
}
?>
<form enctype="multipart/form-data" method="POST">
  <input type="hidden" name="MAX_FILE_SIZE" value="<?php print $upload_max_filesize_inBytes ?>" />
  File: <input name="userfile" type="file" />
  <input type="submit" value="Upload" />
</form>

Test 1: PHP 5.5.10 / Apache 2.4.7

This is a basic Fedora 19 box with standard packages installed. PHP reports Server API as Apache 2.0 Handler.

To get the required setup done I had a .htaccess-file with following contents:

php_value upload_max_filesize "1024M"
php_value post_max_size "1024M"
php_value max_input_time 2

I used time-command from bash-shell combined with a cURL-request like this:

curl --include --form userfile=@800M http://the.box/php/upload.php

Timing results would be:

real    0m7.595s
user    0m1.044s
sys     0m3.259s

That is 7.5 seconds wallclock time to upload a 800 MiB file. The time includes any transfer over my LAN and processing done on the other side. No failures were recorded for the 2 second time limits or memory limits.

Errors would include:

• PHP Warning:  POST Content-Length of 1073742140 bytes exceeds the limit of 1073741824 bytes in Unknown on line 0
• When POST-limit was exceeded
  
• PHP Fatal error:  Maximum execution time of 2 seconds exceeded in Unknown on line 0
• When input processing took too long time

If you are about to go over the 2 GiB limit, please see LimitRequestBody configuration directive for Apache. It is 0 by default meaning 2 GiB. The exact wording from the docs is: "This directive specifies the number of bytes from 0 (meaning unlimited) to 2147483647 (2GB) that are allowed in a request body". However, according to number of sources, the unlimited is spelled as 2 GiB. To go over that, you may need to compile the Apache by yourself.

Warning!
Apache paired with PHP was especially difficult on situations where a HTTP/500 would occur for any reason. The temporary file would NOT be cleaned up immediate after the PHP-script died. The cleaning would occur at the point where Apache worker process would be recycled. Sometimes my temp-drive ran out of disc space an I had to manually trigger an Apache service restart to free up the space. But if you're in server exploiting business and manage to find one that allows large file uploads, it is possible to cause a resource exhaustion for the disc space by simply uploading very large files repeatedly. When upload fails the space is not immediately freed.

Test 2: PHP 5.4.26 / Nginx 1.4.6

To confirm that this is not an Apache-thing or limited to latest version of PHP, I did a second run with a different setup. I took my trustworthy Nginx equipped with PHP-FPM running on a virtualized CentOS. This time I didn't use standard components and used only packages compiled and tailored for my own web server. PHP reports Server API as FPM/FastCGI.

My /etc/php-fpm.d/www.conf had:

php_admin_value[upload_max_filesize] = "1024M"
php_admin_value[post_max_size] = "1024M"
php_admin_value[max_input_time] = "2"
php_admin_value[max_execution_time] = 2
php_admin_value[memory_limit] = 1M

PHP's own ini_set()-function was unable to set any of the values, including those it was allowed to change. I didn't investigate the reason for that and chose to declare all of the required settings in the worker definition.

To get large POSTs into Nginx, my /etc/nginx/nginx.conf had:

location ~ \.php$ {
    client_max_body_size 1024M;
}

Timing results would be:

real    0m16.170s
user    0m1.060s
sys     0m2.854s

That is 16.1 seconds wallclock time to upload a 800 MiB file. The time includes any transfer over my LAN and processing done on the other side. No failures were recorded for the 2 second time limits or memory limits.

Errors would include:

• 413 Request Entity Too Large
• On the browser end
  
• *22 client intended to send too large body: 838861118 bytes
• On the Nginx error log
  

If max POST size was hit.

Conclusions

As found in the net max_input_time and max_execution_time have nothing to do with the network transfer. Both of those limits affect only server's processing after the bytes are transferred.