SEARCH DATAABSE QUESTION

ohenewa

hi guys,
this question might sound bizzaire, but i'm really wondering about something...
here i am, i have created a database search engine for my company's website...
now, i'm wondering, for my database, am i supposed to enter every single word that appears on all the pages of the website? or is there a shorter simpler way of doing it? please help

jalexf

Hi Eliza, two options here depending on how your pages are setup.

1) Are your pages that you want to search regular HTML files? If that is the case, your database search won't work since those pages aren't in the database.

Your choices:

-move these HTML pages into the database and have the database serve them

-find a search engine that will search regular directories and HTML files (fetch.pl is one that I use as it's easily modifyable)

http://www.feartech.com/vv/fetch.shtml

2) Are your pages already in the database?

-You just have to search the right fields.

smarlowe

Save yourself some time and build each page so it can be found from the top of the web site (i.e. no hidden pages) and turn htdig loose on your site. An hour or so later you'll have a professional quality search engine indexing your site with fuzzy matches, soundex matches, phrase searching (beta version only) speling mis-matches and a couple other methods.

www.htdig.org

OR maybe...

htdig.sourceforge.org (sourceforge is still broken in the ftp dept I think)

We use it at work and is VERY nice, and searches are nearly instanteous due to the use a berkely db engine which is WAY faster than most databases, except mysql which is basically similar to a berkley db engine in function and speed.

It's a pretty easy install, and you can customize all the icons and stuff easily to make it match the overall design of your site. you can also segment your searches so people can choose which areas to search before starting.

comstar

There are about 7 thousand ways to do this. Ok, maybe not 7 thousand but close.

Here's a couple fairly easily implementable ideas that may work for you.

Don't even use a database. You COULD just grep the files for the words and do the calculations on your own. This is good if you don't want to worry about your database, etc. It is also good because its 100% current all the time. Its bad if speed is a high priority, because you are going to end up running grep n times where n is the number of words they are searching for. On top of that you'll need to figure out what has the most matches, etc. Perhaps not the best way, but fairly easily implemented.
Use meta-tags in your pages and write a quick crawler that looks for the meta tags in these pages and then enters the url & links it to the meta tag word in the database. You then create a cron entry to run this script every so often (daily perhaps). Of course, you're only going to crawl pages that have a ctime newer than now - time between crawler invocations. This is good because the results are generated MUCH faster. Its bad because the results are only as good as the data gathered the last time the crawler was run.
Do the same as 2 only this time, perhaps add in the content of the pages as well. Same deal, more words in the database.
If your database can support it, you can also store the entire page content in the database. This is something that some sites already do because they archive all their content, etc, and they have web scripts that show the correct content to a user. This is good in that all your search results are now 100% current AND they are fairly quick... not as quick as the indexed words search, but definitely faster than a n*grep. Its bad because it takes more overhead on the database... and more database storage. And then again, lets propose complete disaster - your database crashes. You just lost your complete site unless you've got it backed up.

These are just a few methods that you can use to do this. Its possible not a single one of them may be what you are looking for, but use your imagination and you can come up with a good solution for you. 🙂

Hope that was helpful!

-Philip