Help with search engine

wsh

Hello. I am not very experienced with PHP, but I have some. I am trying to make a search engine and I have 2 questions that I would greatly appreciate help with.

1) How the heck to I parse the RDF files from DMOZ (I dont know XML too well, but even I know that those files are screwed up a little)

2) What is the best method for preforming a search query on a mysql database. Assuming I have stuff in the sql database and someone searches from an html form and the query variable is $q. So it would look like search.php?q=Search+Query. How would I query the database to do the following:

A) Show the most relevent requests first
😎 Write out any matching words in bold
C) Search with the method "any" keywords (so it doesnt look for the phrase "Search Query" - ie: it would bring up the entry:
"When I entered the query into the search field")
D) Limit the results to 30 per page.

I know that is a lot to ask but I already have 300,000+ entries in my MySQL database ... which by the way was a pain in the ass to get out of MS Access... any ideas?

Thank you very much for any help and code you provide. You can email me at boyangel1@shaw.ca if you have any questions.

wsh

If you can answer part of or either of or both of the questions please do so. I really need help and this is confusing.

fuji

a search engine using your db is not something that a person can simply whip up... especially using the information you have provided. what kind of info will you be searching for? how many tables will you need to access? what are the data types for the columns holding the information that you need to access?

With that said, the best way that i can think of searching your db, with the limited info you gave, is to index everything relevant in your db using a simple db lookup table.

id, word, doc_id

where id is the primary key, word is a word... such as the word "terminate" and the doc_id is the document that the word is listed in....

Another way is to loop through a rather large result set, but that will take up resources faster than... [ insert something fast here 😃 ]

As for rdf files, you could do an fopen() then work from there using string manipulation.... but i'm not sure what you want to get out of the rdf file, nor am i sure what you need to do with it once you get it....

Hopefully someone else can help you....

wsh

I thank you for your attempt.

I will give more detail.

I have a database with the following columns:
ID Title Description URL Category Priority

ID - This has the obvious primary key (auto_increment)

Title - The title of the web site submitted to the search engine

Description - A description of the web site submitted to the search engine

URL - The URL of the web site. This would be taken and directly put into a link tag.

Category - The Category the submitted web site belongs to.

Priority - If someone pays for top listings they will have a 1 in this column otherwise the value is 0. I just have the results ORDER BY Priority DESC. However for most relevent first... i dont know how to do this (I can scrap the whole pay for ranking if someone could tell me how to rank by most relevent)

The way I had the search engine (partially) working was to

$res=mysql_query("SELECT FROM dbsearch WHERE Title LIKE(%$q%) OR Description LIKE(%$q%) OR Category LIKE(%$q%)");

As you can imagin a search like that through the 300,000+ entries in my 55MB database takes a while. I have it display results:

while($row=mysql_fetch_array($res)) {
echo "<a href=$row[URL]>$row[Title]</a>";
echo "<br>$row[Description]<br>";
}

Hence I have a simply PHRASE search engine... but I want to make it search containing ANY and ALL options. I also want it to highlite in bold the keywords the user searched from in different parts of the title and description.

I only want to search this one DB and display results from this one DB.

As for the RDF I want to parse it for entries in my search engine (submitted sites) this way I would have a very large search engine.

If anyone can suggest software that ISNT $200 that will parse DMOZ RDF Dumps into MySQL (I can re-write the search engine to comply with whatever the program's format is -if it isnt customizable) I would by 4EVER greatful. However I would prefer a free method of doing so.

THX ALOT FOR ANY AND ALL HELP!

wsh

plz help.. this needs to be done soon and the sooner the help the better. sorry to be impatient and keep posting... but am trying to boost this message to near top so ppl will see it.

wsh

Does anyone know how to parse dmoz rdf files into mysql?

Or how to build the search engine according to the above specs if you can answere any part of the question... plz do.

chriskl

You need to use full text indexing - your 'like' method will be horrifically, horribly slow. Postgres has a tsearch module that will do this, MySQL has FULLTEXTINDEX. Postgres also has ltree (in cvs) which is specially designed for indexing hierarchies and in fact uses DMOZ as its test data. Use them.

To parse XML, compile xml support into your PHP and use the expat parser.

Chris

wsh

Thanks but ... ummm.. ya... uh... i dont have anything available to me but MySQL and PHP (i just have a hosting account on a cluster). I cant compile anything to do with the setup of the cluster servers. If someone could give me an example of the correct way to select as a search query like mentioned above (quicker than LIKE [which takes like five minutes]) My little hosting account really isnt meant for search engines. I should really have a dedicated server.. but I dont want to shell out that much cash. Just want a search engine that searches a large db quickly and displays the most relevent information first.. How do I do this with MySQL and standard PHP settings??? Forget parsing DMOZ I bought a spider.

wsh

HELP... PLZ

cldeans

Chriskl

You mention FULLTEXTINDEX in your answer ... I have been looking in php.net, and cant find a description of this function ... can you elaborate please.

chriskl

Try reading the MySQL manual:

MySQL Manual

wsh - if your hosting provider has at least version 3.23.23 of MySQL running, you can use this.

Chris

wsh

My host has MySQL V 4.0.1A thx! after reading I think I can use this... VERY SMART!

Just wondering though... Can I query like:

mysql_query("SELECT * FROM dbsearch WHERE MATCH (Title, Description, Category) AGAINST ('$query' IN BOOLEAN MODE)");

How can I use the "" operator??? I want to be ably to search for any version of any of the words. For example using +query +word*. How can I parse the initial query (from a form method GET) and change it into words with the correct boolean operators as stated??? Plz provide a funciton or code to make it possible to break down the query and search the db in this fashion... or in the alpha can i not use this??

cldeans

Duh! I think the rain has softened my brain ... thanks for pointing me in the right direction, Chriskl

wsh

My host has MySQL V 4.0.1A thx! after reading I think I can use this... VERY SMART!

Just wondering though... Can I query like:

mysql_query("SELECT * FROM dbsearch WHERE MATCH (Title, Description, Category) AGAINST ('$query' IN BOOLEAN MODE)");

chriskl

No idea - I'm a PostgreSQL man myself!

Chris

wsh

help?

wsh

Noone knows how to do this?