If you do not want the duplicate entries for any purpose then you should not have them on the database in the first place, no point in wasting space on them.
The method of doing this will depend on what you define as a duplicate and if you want the last found duplicate stored or the first.
If the input file is in sorted order then just read it in, store the data and only update the database when the keys change.
If the input file is not in sorted order then perhaps the easiest way is to ensure that the table has a unique primary key set and then try to insert each record in turn, if it fails then try up UPDATE the record instead.
This is not an elegant solution but unless you have a lot of data then it would be a reasonable quick solution.
If you do not have a primary key field then for each record to add you need to read the database first to make sure that there are no matching records currently existsing, if they are then construct and update statement rather than a insert (or contruct a delete followed by an insert if you want).
