Search Techniques

Anon

Hello folks.
I'm redesigning my cheesy personal website (I get about 0 hits per day 🙂 and I was thinking about putting a search in, just for my own experience, really. So, I got to thinking about how to do it, and realized that I've never really come across anything that suggests on good ways to do this in PHP. Maybe you guys can help. What I would want to do is search through some HTML files, (possibly) text files, and definitely some HTML held in MySQL Text columns. I don't know if there is a library out there for this, or an article you can point me to, or a discussion we can have here about the different techniques and the pros and cons of each. Thanks in advance for your time.
Justin

Anon

Hi Justin

I have been implementing something like this for html that is purely database-fed. I was thinking of the hybrid approach and decided it was too messy, especially as I wanted the search engine to be able to highlight the response.

Anyway, Clay Johnson has an article on one way to do this on the phpbuilder site:
http://phpbuilder.com/columns/clay19990421.php3

If you do want to handle existing html, you could kludge together an indexing engine that basically indexes a predefined list of urls via the web-suck feature of fopen() or file(). In pseudocode:

for each $url in url_list
{
fopen($url) save in $str
$array=explode(striptags($str))
for each $word in array{
if (is_significant($word){
insert (url,word) into db.table
}
}
}

stick this in a cron (either using the php executable or a wget) and you're laughing

... and I just found fgetss() which does the striptags step for you.

Andrew

Anon

The "slapping together a search engine" works like a charm, but it builds
a flippin huge table that holds waaaay to much data.

In the end you only want to know how many times a word exists in a url,
so if you use the "slapping" example together with this very nice cron entry
I suggest this little change:

Clean real table
for each file loop{
Clean temp table
Open file.
Fill temporary table with url/keyword combo's
Close file
INSERT INTO real_search_table (URL,word,hits) SELECT URL,word,count(*) FROM temp_table GROUP BY URL,word;
};

that gives you a much smaller search table but still holds the wordcount per URL.