Loop through CSV - NOT inserting duplicates

[sevenfive]

we have a script that opens a CSV loops through the file and inserts the contents into a DB. is there any way to ensure we dont INSERT duplicate records?

thanks in advance

[rane500]

There are a few different options. The easiest is to make whichever field(s) can't be duplicated unique columns in the table, setting the database to reject any duplicate data. (Just make sure you have something in the PHP that will handle the INSERT "failure" gracefully. If it's a combination of fields that can't be duplicated instead of one by itself, you could also do checks against the table before inserting...do a SELECT on whatever combo you don't want duplicated, then do a row result count and make sure it's < 1.

[sevenfive]

ok thanks man!

so we should run the select from within the while loop that processes the CSV???

[rane500]

Absolutely. You'll want to run a SELECT for every line in the CVS file.

Sample program flow:

while(lines are still left) {
    read in the line->line values
    select anything from table where some combination of values exists
    if(rows returned < 1) {
        insert into table values (line values)
        increment to next line
     }
    else {
        increment to next line
    }
}