Comparing large CSV files

[jenn99]

I need to compare two csv files and generate a custom report of the differences. I already have a script written that will do this, but it is processor intensive, and when the two CSV files are large - it just doesn't work.

The way I have it working is basically I loop through each line of one csv and look for a the key in the other CSV (which requires me to loop through that one as well... for each LINE of the first csv.

So you can imagine.. if the files are both 100 lines... the script will have to loop 100x100 times!!! (The reason I can't just compare line to line - is that there may be lines missing/added between the two...)

Is there a better way?? Is it worth making a temporary database out of one of the CSVs so that I can just loop through ONE CSV file and do a simple QUERY to find the key in the other?

I know there are applications that will do a file comparison and diff report, but that's the other issue, I need to generate a very customized report from the comparison.

Any help/suggestions people have would be GREATLY appreciated! I'm at a total loss. 🙁

[onion2k]

If its only 100 lines in each, read the keys from one into an array and then loop through the other one doing an "in_array()" comparison with the id and the array of ids from the other file.

10,000 comparisons isn't much.. I've written scripts that do tens of millions of calculations in the past.. PHP can cope without any problems.

[jenn99]

Actually the smallest file I'll be working with is over 3000 lines (hence my concern... 3000x3000 loops. Yikes)

But! Your suggestion helped a lot. Since the key can show up in more than one place, I actually used array_keys() to get all the occurances...

In any case, it seems to handle the 3000 line file okay... hopefully it won't choke as the files get bigger!!

Thanks again!