How to read a large text/csv file

scraff

Hi Everyone,

Ive got a system set up to transfer data to a database. In my admin area, I have a textbox into which I copy and paste the contents of a "|" deliminated CSV file.
"|" seperates each field, and a return "\n" each line is a new row.

The next page takes the information, and puts the data into the database, each row at a time, and each field in each row, and goes through the whole text until all the records are in the database.

But Ive come across a problem that what the file is too big (when I say file, I mean the text ive copy and pasted into the textbox from the file) when this is too big (sometimes 24mg) it takes forever, then just doesn't add any of the records into the database.

Is there a better way to do this?
Is there anyway of uploading the CSV file and use PHP to read the file?

Thanks

johanafm

You can upload your file any way you like. ftp, scp or using an html form with <input type="file"...>. If you go with the html solution, make sure that you accept large enough post data size and big enough file size (google "list of php ini directives" to get to the appropriate place in the php doc).

Once the file is on your server, use [man]fopen[/man] and [man]fgetcsv[/man].

scraff

Ok thanks that should do it. I just thought fopen and fgetcsv was for files already on the website/server. If there a way of reading a file using

<input type="file"...>

without saving it to the site?

bradgrafelman

scraff;10989550 wrote:
Ok thanks that should do it. I just thought fopen and fgetcsv was for files already on the website/server.

Well, fopen() isn't limited to just opening files, but yes, of course the server has to have some way of accessing the resource you're telling it to open... hence why johanafm was suggesting you upload the file to the server.

scraff;10989550 wrote:
If there a way of reading a file using
<input type="file"...>
without saving it to the site?

So you want the server to parse data that you don't want to send it? Er... well it should be quite obvious why that's never going to work. What's wrong with using a file upload?

scraff

Lol I do want to send it, what I mean is I want to send the data to the server, but without actually having to save a file physically. Temp would be fine

bradgrafelman

Alright, so then just use a file upload but don't call [man]move_uploaded_file/man - just read the file from it's temporary location and let PHP delete it when your script is done executing (or manually [man]unlink/man it yourself, if you'd like).

sneakyimp

Some good reading on file uploads here. A few key ideas:
be sure to set the enctype in your HTML <FORM> tag to 'multipart/form-data' and make sure method=post

<form enctype="multipart/form-data" action="some_file.php" method="post">

* in your php script, some_file.php in this case, you should find information about the uploaded file in the superglobal array, $FILES. This array is first indexed by the name of your input and secondarily by the details about each file. The uploaded file will be stored at a temporary location you can find in $FILES['your_input_name']['tmp_name']
Make sure you validate the uploaded file as much as you can to avoid reading some virus-infested file or exploit. Keep in mind the uploaded file could be almost anything under the sun.
Avoid trying to read too much data from the file at once -- you should do this to protect your server from buffer overflow exploits.
Read the file one line at a time and handle each line independently. Don't try to store the entire file or some giant data structure parsed from it or you'll just run out of memory again.
when you are finished, close the file, destroy any references to it, and delete the temporary file to conserve memory and disk space.