Hello all,
I have large text files (up to 1 million lines), field seperated for easy import into a MySQL database with 3 fields of data. Prior to import, I want to count duplicate records in the file row to row based on a given field.
I know that I can ignore duplicate entries through MySQL in the import process, but it will ignore entries that I have existing in the table, offsetting the final count of duplicates. I just want to find out how many duplicates there are in the file based on the field.
I was considering using arrays to help determine duplicates, but that would be a mighty long array, with over a million lines in the file. Any ideas???
Thanks,
-Alex
