[RESOLVED] Would this slow down my script?

mrbaseball34

I have a script that parses large CSV files. It has a sprinkling of "progress messages" in it, two per parsed line. I am using output buffering and using ob_flush to flush the buffer at various times.

During the initial processing, it is very fast, importing 6700+ rows in the first 10 minutes of processing, but 4 hours and 35,000 rows later, it is down to doing about 650+ per 10 minutes.

I am making sure that I am unsetting my variables and using mysql_free_result where I actually return a result from the DB.

After the 4+ hrs Firefox is using over 200MB of memory per task mgr.

I have made mods to the script to only output anything if a flag is set but not sure if that is all I can do to make this thing run better.

I have another question, I can import the CSV using LOAD DATA INFILE and then iterate through the records to import each record but is that any faster than importing the CSV line by line?

dagon

sounds like something that should be running from the command line with out output.

mrbaseball34

I can't because it is to import data into my web store.

dagon

mrbaseball34;10961099 wrote:
I can't because it is to import data into my web store.

i don't see why that would be an issue, you still don't need to see what it is doing do you?

mrbaseball34

if I run it locally and want to see output, yes, I would.

dagon

you say it takes hours, what are you doing sitting there watching it for hours? what's the point?

mrbaseball34

The point is that sometimes I take smaller files for testing and use them and require output to see if they are working correctly.
I test different column arrangements and our "algorithm" to make all the different column arrangements work with our system.

mrbaseball34

I'm going to mark this as resolved since it is not getting constructive feedback.