Search Engine

Arc

I was wondering how some sites do search engines. A message board is fairly easy because everything is in the database and you can just search the title and body for matching words, but what about a site like microsoft where you type in a program name or whatever and it gives you the link to that page.

For instance i am creating a site for a record company that is Database driven for the most part, but some of it will be hard coded and not really dynamic. So how would i create a search engine that would bring up every page that had that word in it?

For example, if someone typed in Garth Brooks i would want to bring up a list of pages
and a small snippet of the text from that page that realtes to garth

www.recordcompany.com/news.php
Garth Brooks appears in Nashville on a spur of the moment trip...

www.recordcompany.com/music.php
Hear Garth Brooks latest release...

www.recordcomapny.com/artists.php
Welcome to garth Brooks artist page....

etc..

How is this possible? How do most sites do it? Thanks!
😃

superwormy

Lots of sites use another websites API... like Googles or a third party like www.atomz.com

But if you want to make your own... setup three tables:

table_Words
ID,word,metaphone

table_Link
ID,word_ID,page_ID

table_Pages
ID,page,title,descrip

Feed a little script every URL you want to index, have it first strip the HTML, then insert ( or dont' if word is alreayd there ) words into your word table, the page into your page table, and links between the two into your link table.

When someone runs a search, find the word_ID, then find the links with that word_ID, then select all the pages that are appropriately given to you by that link table.

Arc

Hmm well that might work for a static site, but most of this site is dynamic and only a small portion is static. Since most of the data from the site is in a DB i can't very check the pages themselves and put that into a DB. I basically have to check the DB for the words, but then i have no idea exactly what page the data is going to be on as the data is stored in categories like News,Featured Artists etc.. but there may be several pages in each category. hmmmm... Plus this method totaly leaves out any static pages. Even the static pages change from time to time but they are hard coded and not DB driven.

How do you use google? Do you have to pay for that?

superwormy

Um... maybe you didn't understand my post... my explanation was pretty much EXACTLY what Google will do, only Google is a lot better at it and has 1,000 computers doing it 24 hours a day, 365 days a year. :-)

Every page is accessible by users right? I mean, if parts of it are only accessible via form submits ( POST method ) then you've already pretty much screwed yourself on those pages, because Google won't find those either.

But as long as you have at least one <a href="link.php"> link to each page you want to be searchable, you can write a spider to find and index all those.

Another thought ( if I'm missing something ) woudl be to index pages that are static like I said above, and then index the content in the database with a similar script, only its not grabbing pages to index, its grabbing stuff from the database.

Arc

Why would i need to set up tables though? Google doesn't store all these sites in a database does it? I mean it does a new search of each page every time you request it right? Otherwise it could be showing very old information. So anyways... exactly how does one create a "spider" to search ones own site? I could hard code the url's to each page for it to search on then have it load those pages,which is the only time the data is there becasue it pulls from a DB, then search each page for the "words" then give the list.

Only thing is i have no idea how to search a page like that thru the URL:/

superwormy

Why would i need to set up tables though? Google doesn't store all these sites in a database does it? I mean it does a new search of each page every time you request it right?

... You're Kidding Right? ...

OF COURSE Google stores all the data in a database! Can you IMAGINE how long it woudl take Google to search 3 BILLION WEBPAGES EVERY TIME SOMEONE TYPED IN A QUERY???

That woudl be insane. It would take like a day and a half to do a single search :-)

Hehe, don't mean to be harsh, but YES, Google definitally uses some form of database. It's actually a very distributed filesystem sorta database, but its a database. Like 99% of all web content is reasonably static, and they recrwal and reindex their entire set of pages every month or two. 99% of those pages probably dont' even change.

See these PHP functions to read pages over HTTP connection:

www.php.net/fopen
www.php.net/fread
www.php.net/fsockopen
www.php.net/fclose

You can find URLs by writing a regular expressoin ( www.php.net/eregi_match ) or using some fancy string matching ( find string position of href=", then read until you find " or a space )