Need to find "similar" rows in MySQL

[arewhyfour]

I'm pulling in feeds from multiple sites that sometimes have duplicate information between sites. I'm inserting them all into a database, here is the information I'm inserting:

(Feed 1) # Ric Flair On His Loyalty To WWE, His Greatest Opponent
(Feed 2) # Ric Flair On His Loyalty To WWE, Says Sting's Not His Greatest Opponent, His Future

(Feed 1) # WrestleMania Festivities, Austin/Hatton, WWE Teams With NHL Team, More
(Feed 2) # WrestleMania Events, Austin/Hatton, WWE Teams With NHL

(Feed 1) # Consequences Creed's Return To TNA, Tonight's TNA House Show, More Notes
(Feed 2) # Consequences Creed's Return, TNA House Show, More Notes

As you can tell, the titles are quite similar, but yet very different.

I need some way of running a sql query that'll clean this duplicate information up.

[NogDog]

I do not think there will be an easy solution to that: you're starting to get into the realm of "fuzzy logic" and artificial intelligence. The only possibility I can think of would be to do something with MySQL's Natural Language Full-Text Searches.

[dagon]

as well as the above have a look at

similar_text()
levenshtein()
soundex()