Code Breaks with large files

edziffel

Have written test code to upload csv files to insert into sql. The code works for small files and at least up to 1000 records or a 374K file which was made with just the first 1000 records of the original file. At some point it breaks and fails giving me the errors:

Message: Undefined index: fileupload (my name for the file)

Message: fopen() filename cannot be empty

All the code does at the moment is to upload a csv file to a temp folder called with $_FILES['name']['tmp_name'] then fopen and then iterated to display the contents of the array to the monitor. Worked fine with smaller files but...

Have to upload as many as a million records or a 92meg file. Are there config settings that need to be changed? Are there other issues that need to be considered?

Other info that might make a difference.

Files open in Excel 2010 and then pasted to new worksheet as values to clean up if needed as files come from outside source then saved as csv. Checked in note pad and the data looks clean enough, it does work in chunks.
Running a quad core at 3.2ghz with 4gig 1333 men on a linux install using lamp, if that makes a difference, which at some point do doubt it does.

johanafm

Check php.ini for values of "post_max_size" and "upload_max_filesize". If you exceed filesize, but not post size), you should have an entry in $_FILES for which error is UPLOAD_ERR_INI_SIZE. If you exceed post_max_size, I believe you'd get the error you are seeing here, i.e. no file at all. Obviously post_max_size should always be bigger than upload_max_filesize since there will be more post data than just the file.

# Was the submit input "submit" clicked?
if (isset($_POST['submit']))
{
	# Is the file input field "file" even supplied?
	if (isset($_FILES['file']))
	{
		# max file size if you need to deal with smalle file sizes in this case
		# than what php.ini allows.
		$MAX_SIZE = 4096 * 1024;	# 4MB


	# And was the file upload ok?
	if ($_FILES['file']['error'] === UPLOAD_ERR_OK)
	{
		# upload went ok. do what you want

		# If you have a smaller size limit for this particular file
		# than what is spceified in php.ini, you'd check file size here as well
		if ($_FILES['file']['size'] > $MAX_SIZE)
		{
			# inform user of max file size
			# Use $MAX_SIZE to inform them of max file size.
		}
	}
	elseif ($_FILES['file']['error'] === UPLOAD_ERR_INI_SIZE ||
			$_FILES['file']['error'] === UPLOAD_ERR_FORM_SIZE)
	{
		# inform user that the file was too big
		# Use $MAX_SIZE to inform them of max file size.
	}
	elseif ($_FILES['file']['error'] === 'other error code you wish to handle specifically')
	{
		#...
	}
	else
	{
		# all other errors.
	}
}
else
{
	# nothing to do. 
}
}
else 
{
	# nothing posted
}

Also, always make your code able to handle all conditions gracefully, by never assuming that you have valid data to work with. Or that you have any data for that matter.

edziffel

Wrote own code no check for file size as I knew it was going to get large, with only specified users so no public access. As stated above cleaned data using only a value paste ms office 2010. Also checked samples visually in addition to being able to use selected parts or segments of records just fine.

Found on line that you need to change the default php setting which limit file upload size to 2meg.

Working now. Just one problem. It was taking a long time to print to screen more than a couple of minutes. Acttually since I use a multiple machine set up with a kvm switch box, I'm pretty sure it is still trying to print even now. However, since I have the codes set up to parse every step and show variable and array values to screen before proceeding to the next step, I know the file uploaded and is being processed. Was concerned initially like "Oh no! wheres my scrolling view!!! What is happening!!!!". Then I did the math. One record displays at about rounding down 6 inches of screen space. So keeping it simple that means that every 10 thousand records is roughly a mile long on screen. So for a million records that means my virtual scroll is going to be more or less 100 miles long LOL, LMAO. That could take a while right? No doubt something else is going to break first on that one. :-)