search method

[jpr]

Hey, currently the method I use for searches run on my website are as follows:

1. Take search string from user input
2. Search string as a whole first exactly
3. Remove minor words
4. Search individually main words by using the "LIKE" function in mysql

Is this a good/bad way of doing this? It is a lot of messy code. Not to mention the long list of words that I consider minor words that have to be removed. Any suggestions? Thank you!

Jason

[lazzerous]

I use common words filtering for my search function as well. The way I see it, if Google does it, it's prolly a good idea.

Here's the function I use for the filtering, along with referenced functions:

<?
// function common_words_filtering
// this is the first step in the search process
// the the user's query, $q, and remove any common words that would ruin the relevence of the search results
// return
//  - $q
//    - the new query without common words
//  - $words_removed
//    - an array of the words removed to notify the user
// these are returned in an array
// 
// to properly call this function:
// list($q, $words_removed) = common_words_filter($q); 
function common_words_filter($q)
	{
	// kick entire query down to lowercase
	$q = strtolower($q);

// weed out common words from search
$commonWords = array("a", "and", "am", "as", "in", "or", "the", "for", "is", "as", "was", "why", "be", "been", "are", "i", "it", "to", "i'm", "i've", "you", "your", "yours", "you've", "you're", "their", "they", "them", "after", "although", "and", "as", "because", "before", "but", "cos", "except", "for", "if", "nor", "once", "or", "provided", "since", "so", "than", "that", "though", "unless", "until", "when", "where", "whereas", "whether", "while", "whilst", "a", "all", "an", "another", "any", "both", "each", "either", "enough", "every", "few", "former", "half", "her", "his", "its", "it", "latter", "less", "little", "many", "more", "most", "much", "my", "next", "no", "our", "own", "same", "several", "some", "such", "that", "the", "their", "these", "this", "those", "what", "whatever", "which", "whose", "your", "to", "aye", "no", "well", "yeah", "yes", "can", "could", "may", "may", "might", "must", "ought", "shall", "should", "used", "will", "would", "about", "above", "according to", "across", "after", "against", "along", "among", "amongst", "around", "as", "at", "by", "for", "from", "in", "like", "near", "of", "onto", "to", "towards", "under", "until", "up", "upon", "via", "with", "within", "without", "worth", "anybody", "anyone", "anything", "everybody", "everyone", "everything", "he", "her", "herself", "him", "himself", "his", "i", "it", "itself", "me", "myself", "no-one", "nobody", "none", "nothing", "one", "other", "ourselves", "plenty", "she", "somebody", "someone", "something", "them", "themselves", "there", "they", "us", "we", "who", "whom", "you", "yourself", "have", "been", "be", "not", "looking", "how", "do", "find", "line", "get", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z");
// array_walk($commonWords, 'test_print');

// use spaces and commas to split the query string into an array
$q_ar = search_terms_array($q);

$new_query = "";
$words_removed = array();
for ($i=0; $i<sizeof($q_ar); $i++)
	{
	// remove any remaining whitespace from search term
	$q_ar[$i] = trim($q_ar[$i]);

	$term_used = false;
	for ($j=0; $j<sizeof($commonWords); $j++)
		{
		if ($q_ar[$i] == $commonWords[$j] && $term_used == false)
			{
			$words_removed[] = $q_ar[$i];
			$term_used = true;
			}
		}
	}
}


function search_terms_array($search_terms)
	{
	$search_terms = preg_split("/[\s,-]+/", $search_terms);
	$final_terms = array();
	$count = "0";
	for ($i=0; $i<sizeof($search_terms); $i++)
		{
		if ($search_terms[$i] != "" && $search_terms[$i] != " " && $search_terms[$i] != NULL)
			{
			$final_terms[$count] = $search_terms[$i];
			$count++;
			}
		}

return $final_terms;
}


?>